Senior Data Engineer (AI/ML and AWS Cloud)

About this role

Company Overview

Pantheon Data (a Kenific Holding company) is a private, small business based in the Washington, DC, area. Pantheon Data was founded in 2011, initially providing acquisition and supply chain management services to the US Coast Guard. Our service offerings have grown in the past ten years, including infrastructure resiliency, contact center operations, information technology, software engineering, program management, strategic communications, engineering, and cybersecurity. We have also grown our customer base to include commercial clients. The company has used this experience to expand our service offerings to other agencies within the Department of Homeland Security (DHS), the Department of Defense (DoD), and other Federal Civilian Agencies.

Position Overview

We are seeking a Senior Data Engineer to design, build, and optimize the data foundations for our next-generation Generative AI applications. This role is focused on architecting the Data Enrichment and Vectorization pipelines that power Large Language Models (LLMs). You will be responsible for the end-to-end lifecycle of data, from ingestion in AWS to serving high-context, enriched datasets to AWS Bedrock.

Responsibilities

• **LLM Data Pipelines:** Design and implement scalable data ingestion and transformation pipelines specifically for RAG (Retrieval-Augmented Generation) architectures.
• **AWS Bedrock Integration:** Operationalize LLM workflows using AWS Bedrock, managing model invocations, and embedding generation.
• **Data EnrichmentQuality:** Develop advanced Python-based processing jobs to clean and enrich unstructured data with metadata to improve LLM retrieval accuracy.
• **Vector Database Management:** Architect and maintain vector stores (e.g., OpenSearch Serverless or Postgressql pgvector) to support efficient semantic search.
• **Cloud Architecture:** Leverage core AWS services (S3, Glue, Lambda, Step Functions) to build resilient, automated data w...