Lead Data/AI Engineer

About Insurance Insider

Insurance Insider is the leading provider of insights and analysis for the world's top insurers, distributors, service providers, and investors.

Since 1996, we've been helping clients uncover new business opportunities and protect against risks through our exclusive news, deep analysis and actionable insights on the insurance market. Our coverage extends across the London market, global (re)insurance market, insurance-linked securities market and US property and casualty market.

About the Role

We are looking for a Lead Data/AI Engineer who writes real code, builds robust pipelines, and takes
ownership of data infrastructure from end to end. You will be comfortable across Python, TypeScript,
and/or Node.js, and will use AI tools to work efficiently and ship high-quality solutions.
This role spans data engineering, analytics configuration, and content integration, giving you broad exposure across the stack without losing depth in the engineering fundamentals.
You bring a few years of solid experience, have strong opinions about good pipeline design, and can operate independently while collaborating closely with product, editorial, and commercial teams. Ideally you will have worked in a data business, one that sells proprietary data directly to customers and/or distributes it via different
channels (APIs, MCP, dashboards), and you understand the engineering and commercial implications of that model. This is also a people manager and mentor for a small team of developers, setting technical direction and raising the bar for how the group builds.

Key Responsibilities

Data Engineering & Pipeline Development

• Design, build, and maintain data pipelines and orchestration workflows using Databricks, Python, and/or TypeScript/Node.js – covering ingestion, transformation, scheduling, and monitoring.
• Write clean, well-structured scripts and services to ingest, process, and transform data from APIs, databases, and third-party sources – using Python for data work and Node.js/TypeScript where appropriate for integrations or lightweight services
  

Technical Leadership & People Management

• Line-manage, mentor, and grow a small team of data engineers – running 1:1s, supporting career development, giving clear feedback, and helping the team do the best work of their careers.

• Set technical direction and standards – making sound architectural decisions, defining how the team works, and balancing speed of delivery against long-term maintainability
  
  

Billing Systems

• Define requirements for billing, subscriptions, invoicing, and revenue workflows in partnership with Product and Finance

• Lead the end-to-end build-out of a new billing system – from discovery and vendor/build evaluation through to delivery and adoption

• Integrate a usage-based billing system such as Orb or Metronome – including designing the usage logging layer that captures metering events (API calls, data queries, MCP tool invocations, records returned) from pipelines and services, and making that data reliably available to the billing platform for customer charging and entitlement enforcement
  
  

Analytics & Insights Configuration

• Configure and maintain dashboards, reports, and analytics tools to surface insights from content and data – connecting pipeline outputs to BI tools and keeping metrics accurate and up to date.

• Work closely with editorial, product, and commercial teams to understand their data needs and translate them into practical metrics and visualisations.
  
  

Content & Data Integration

• Connect content assets with structured data using scripted pipelines – enabling smarter search, tagging, categorisation, and intelligence features across our platform.

• Identify new data sources and APIs, scope their integration, and build the connectors needed to operationalise them – fast.

• Build and maintain MCP server integrations that expose Insurance Insider’s proprietary data to AI agents and client workflows.

• Design and implement data-sharing infrastructure – enabling structured, permissioned delivery of datasets to customers via APIs, data-sharing platforms, or direct integrations with client systems.
  
  

AI-Assisted Development

• Leverage AI coding tools (e.g., GitHub Copilot, Claude, Cursor) to write, debug, and iterate on data scripts, pipelines, and orchestration logic quickly and independently.

• Stay curious and self-directed – experiment with new tools and approaches, and bring practical solutions to the team without waiting for perfect resources or large teams.

• Develop working prototypes for AI-based products and/or AI internal tools, using AI coding tools to create quickly and iterate
  
  

Stakeholder Collaboration

• Work across teams – product, editorial, commercial, and leadership – to understand data needs and surface the right insights at the right time.

• Communicate data findings clearly to non-technical audiences, making complex outputs accessible and actionable.
  
  

Operational Excellence

• Monitor data quality across pipelines and outputs, flagging issues proactively and implementing fixes to maintain reliable, trustworthy data.

• Continuously improve data processes, tooling, and documentation so that the team’s data assets are scalable, maintainable, and easy to build on.
  
  

Required Experience

• 5–8 years of hands-on software or data engineering experience, with strong proficiency in Python and working knowledge of TypeScript or Node.js – comfortable writing production-quality code, not just scripts.

• Proven experience designing and running data pipelines and orchestration workflows in production – with solid understanding of scheduling, dependency management, retries, alerting, and data quality patterns.

• Solid experience with Databricks, Spark, or a comparable cloud data platform; confident with SQL, Delta tables, and both notebook-based and job-based pipeline architectures.

• Experience connecting data pipelines to analytics or BI tools (e.g., Power BI, Looker, Metabase) and maintaining the data models that underpin them.

• A strong engineering foundation combined with a generalist mindset – able to move between data engineering, analytics, and integration work and make sound technical decisions independently.

• Experience working at a business that sells data as a product – whether through direct licensing, API access, or data sharing platforms – with a solid understanding of the data product lifecycle, access control, and the commercial sensitivities of proprietary datasets.

• Some experience managing or leading other data roles – whether line managing analysts or engineers, setting technical direction for a small team, conducting code reviews, or mentoring junior colleagues. You don’t need to have run a large team, but you should be comfortable owning quality and direction across the data function.

• Demonstrated use of AI coding assistants (e.g., Copilot, Claude, Cursor) as a genuine multiplier – not a crutch. You write good code, use AI to go faster, and know the difference.
  
  

Preferred Qualifications

• A degree in computer science, engineering, or a related technical field – or demonstrably equivalent experience. We care more about the quality of your engineering than where you studied.

• Experience with orchestration tools such as Airflow, Dagster or Databricks Workflows; be confident building REST API integrations, webhook consumers, or event-driven data patterns in Node.js or Python.

• Hands-on experience with MCP (Model Context Protocol) – building or consuming MCP servers, understanding tool definitions, and thinking through the access, security, and versioning challenges of exposing commercial data via this protocol.

• Familiarity with usage-based billing platforms such as Orb or Metronome – including how to design and instrument a usage logging layer that reliably captures metering events (such as API calls, data queries, MCP tool invocations, and records returned) from pipelines and services; how those events are ingested and processed by a billing platform; how usage is tracked against customer entitlements and plans; and how billing data surfaces in finance, product, and customer-facing reporting.
  
  
  

What We Offer

• Flexibility with true hybrid working – expected to be in the office 1-2 days a week.
  
• 25 holiday days per year, plus your birthday off.

• A collaborative and mission-driven culture.

• Opportunities for professional growth and development.
• Competitive compensation and benefits package.