Senior Python Engineer (Remote)

Job summary

This role involves evaluating AI coding agents by creating realistic developer environments and tasks to assess their performance on real-world developer challenges. The position is project-based and fully remote.

Qualifications

• Over 8 years of experience in software development.
• Proficient in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
• Experience in writing functional and integration tests.
• English proficiency at B2 level or higher.

Responsibilities

• Develop realistic developer environments that simulate a virtual company with a codebase, infrastructure, and context.
• Design tasks based on intermediate states of these environments, defining what constitutes a 'solved' task.
• Write tests to verify the solutions provided by AI agents, ensuring a balance between strictness and leniency.
• Iterate on tasks and tests based on quality assurance feedback, refining them to ensure fair evaluations.

Skills

• Strong understanding of AI model limitations and capabilities in coding.
• Ability to create challenging tasks that effectively evaluate AI performance.

Education

• Not specified.

Tools

• Python, FastAPI, JavaScript/TypeScript, Docker, Postgres, Kafka, Redis.