For the Applied AI Software Engineer role at Shepherd, success in production is defined by metrics that balance model performance with operational impact. Key technical KPIs include Inference Latency and System Throughput, which ensure the underwriting engine remains responsive. Accuracy and Reliability are critical; this is measured through Retrieval-Augmented Generation (RAG) metrics like hit rates, precision, and recall, alongside semantic consistency and hallucination rates in model outputs.

Furthermore, success is measured by End-to-End Workflow Efficiency—specifically, how often AI-driven submissions reach a final decision without human intervention. Finally, Evaluation Framework coverage is paramount; successful deployments require robust monitoring of model drift, cost-per-inference, and the "human-in-the-loop" override rate to ensure continuous optimization.

At Shepherd, we balance model innovation with reliability by treating AI as a first-class citizen within our software development lifecycle. We mitigate the non-deterministic nature of LLMs by implementing rigorous evaluation frameworks and automated monitoring to supervise model outputs in production.

We prioritize a "human-in-the-loop" architecture, where AI agents handle routine tasks while complex risk evaluation remains auditable. By leveraging RAG architectures and reliable vector database retrieval, we ground model decisions in structured, real-world construction data. This ensures that new innovations—like autonomous underwriting—are deployed incrementally, allowing us to maintain the strict consistency, accuracy, and technical reliability required to modernise high-hazard commercial insurance.

The most transformative trend in insurance underwriting is the shift toward fully autonomous, agentic workflows. By moving beyond static forms, Shepherd is leveraging LLMs and RAG architectures to ingest unstructured data from diverse sources—such as construction site sensors and project management software—to perform real-time, automated risk assessment.

Furthermore, the integration of multi-step reasoning systems allows AI to handle complex edge cases that previously required human intervention, enabling "email-in, price-out" capabilities. This shift from manual, document-heavy processes to AI-native infrastructure allows firms toprice risk with unprecedented speed and accuracy, turning underwriting from a reactive, fragmented process into a high-leverage data advantage that compounds safety and operational efficiency.

At Shepherd, we bridge the gap between AI outputs and human underwriting through an integrated, agentic workflow. We facilitate feedback loops by designing AI-powered tools that extract structured insights, enabling underwriters to review and interpret data faster. As underwriters interact with these systems—validating pricing models and assessing risk outcomes—their actions serve as critical data points for model refinement. We prioritize building robust infrastructure for monitoring and evaluation, ensuring that human intervention in the "last mile" of the underwriting process directly informs the performance and reliability of our agents. This continuous human-in-the-loop approach ensures our pricing and evaluation engines evolve, compounding safety, speed, and accuracy across our insurance platform.

Shepherd’s AI-native model transforms product roadmaps by treating AI as a core architectural layer rather than an auxiliary feature. Unlike legacy insurers reliant on static documents, Shepherd integrates real-time data from construction partners like Procore and Autodesk directly into its underwriting workflows. This technical approach allows the Applied AI team to prioritize autonomous "agentic" systems—such as automated submission processing—that move from intake to quotation with minimal human intervention. By embedding LLMs and RAG architectures into high-hazard risk assessment, Shepherd shifts the focus from manual data entry to building scalable, production-grade infrastructure that translates complex, unstructured construction data into precise, automated pricing models and smarter underwriting decisions.