Senior Python Engineer, Vice President

We are seeking a Senior Python Engineer to fill the role of ‘Applications Development Technology Lead Analyst’ on the XiNG large scale distributed calculation platform (XiP).

In this role you will design, build, and maintain the next generation of the XiP platform. You will be working on a Greenfield build of a major new component. The current generation of the product is responsible for 500 million risk & suitability calculations at Citi every day. The next generation of the product is expected to double the customer base and bring in huge new workloads.

As an engineer working on XiP, you will work on a platform which daily; coordinates hundreds of thousands of pods, consumes tens of thousands of compute cores, utilizes hundreds of TB of memory, and distributes hundreds of millions of calculations.

You will work closely with software engineers, DevOps, and our customers to ensure reliable, secure, and scalable operations of the platform.

Responsibilities:

Take part of the shared responsibility for the long-term strategic direction of XiP:

Apply your experience and skills to develop the future versions of the calculation engine. This means making it faster, utilize resources better, and scale to even greater sizes. This means adding new features without impacting performance of the platform in undesirable manner, small changes multiplied by millions of calculations have a high cost. This means solving hard problems like transferring small amounts of data to a huge number of machines.

Serve as advisor or coach to team members, allocating and overseeing work:

Mentor and guide mid-level developers, providing technical advice, code reviews, and career development support. Allocate tasks based on skill sets and project priorities, ensuring balanced workloads and timely delivery. Foster a collaborative and growth-oriented team environment, encouraging best practices and continuous learning.

Implement the new generation of Citi's Risk Calculation Engine:

Participate in the design, development, and implementation of the next version of the XiP Risk Calculation Engine, focusing on understanding and fulfilling the needs of new platform customers.

The current generation of the platform regularly parallelizes 250,000 hours of compute into a single 90-minute execution. The new platform needs to maintain this performance while enabling new workloads.

Operationally support the optimization of Risk calculations by examining and providing SME level insight into the distributed execution of extremely large calculations:

Monitor and analyze the distributed execution of large-scale risk calculations, identifying performance bottlenecks and inefficiencies. Provide subject matter expertise to guide operational improvements, leveraging deep technical knowledge to recommend and implement optimizations that enhance calculation speed and resource utilization.

Plan, coordinate, and execute strategic changes to the engine to reach the next scaling milestone:

Develop and manage strategic initiatives aimed at scaling the Risk Calculation Engine to meet future business demands. Coordinate with stakeholders to ensure that all scaling efforts are aligned with customer needs.

Provide specialized customer support to optimize and improve large, distributed calculations.

Act as a primary point of contact for clients to address issues and optimize inefficiencies that arise during distributed workload execution.

Qualifications:

5+ years of professional software development experience with Python.

Solid experience with Python frameworks (e.g. Flask, FastAPI, Pydantic ).

Experience with python packaging and environment managers (pip, conda, virtual environments)

Experience with async programming in python (e.g. multiprocessing, subprocess, concurrent.futures)

Exposure to cloud platforms (e.g., AWS, Azure, GCP) and containerization (e.g., Docker, Kubernetes).

Experience designing, building, and maintaining large-scale, distributed systems.

Experience with RESTful API design and implementation.

Exposure to NoSQL databases (e.g., MongoDB, Cassandra) is a plus. Experience with unit testing, integration testing, and test automation frameworks (e.g. Pytest, unittest).

Experience working in Agile/Scrum development environments.

Experience with CI/CD pipelines and DevOps practices is a plus.

Education:

Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.