Compensation & Location
Location: Torrance, CA
Pay Range
$52.00 – $59.00 per hour (based on experience and internal equity)
Schedule: Full-time, contract; long-term opportunity
Work Type: Hybrid (4 days onsite, 1 day remote)
Benefits
Health, Vision, Dental, Life Insurance, 401k, Continuing and on-demand education
📍 Must be willing to work onsite in Torrance, CA (LA County)
✅ Sponsorship is available for this role
- ‼️ ONLY W2
- NO CORP to CORP
- NO 3rd Parties
Position Summary
We are seeking an AI Business Data Analyst to support AI-powered customer experience products and digital transformation initiatives within a large enterprise environment.
This role will focus on monitoring, evaluating, measuring, and optimizing production Generative AI and Conversational AI applications while providing actionable insights that drive product strategy, roadmap decisions, and continuous improvement efforts.
The ideal candidate combines strong business analysis, product analytics, and data analytics experience with a practical understanding of Generative AI, Large Language Models (LLMs), Conversational AI, Retrieval-Augmented Generation (RAG), and AI evaluation methodologies.
Key Responsibilities
- Establish, monitor, and report on KPIs related to AI product performance, customer experience, operational efficiency, and business value
- Analyze production AI application data including conversation logs, usage patterns, customer feedback, quality signals, escalation trends, and operational metrics
- Identify opportunities and risks related to response quality, accuracy, hallucination risk, retrieval relevance, content drift, customer satisfaction, and task completion
- Design and support AI evaluation frameworks including quality reviews, benchmark testing, evaluation rubrics, human-in-the-loop reviews, and LLM-as-a-Judge methodologies
- Build dashboards, scorecards, and executive reporting to communicate AI application health, trends, risks, and improvement opportunities
- Translate analytical findings into actionable product requirements, backlog items, user stories, acceptance criteria, and release validation plans
- Conduct exploratory analysis of customer behavior, conversation topics, AI-assisted journeys, and feature usage patterns
- Support experimentation and A/B testing for prompts, workflows, retrieval strategies, response patterns, and AI feature enhancements
- Participate in Agile ceremonies including backlog refinement, sprint planning, sprint reviews, demos, and retrospectives
- Support UAT, release validation, post-release monitoring, and performance measurement for AI product enhancements
- Partner with Product Owners, AI Engineers, Data Teams, QA, CX stakeholders, and business leaders to support AI product delivery and optimization
- Translate technical AI findings into clear business insights for both technical and non-technical audiences
What You'll Be Working On
- Generative AI and Conversational AI customer experience applications
- Large Language Model (LLM) performance monitoring and evaluation
- AI-powered customer service and digital experience initiatives
- KPI development, dashboard creation, and operational scorecards
- AI quality monitoring, hallucination detection, and response evaluation
- Product analytics, roadmap planning, and customer journey analysis
- Prompt engineering evaluation, retrieval optimization, and RAG performance analysis
- Agile AI product delivery and continuous improvement initiatives
Required Skills & Experience
- Bachelor's degree in Business, Analytics, Statistics, Data Science, Computer Science, Information Systems, Economics, Marketing, Cognitive Science, or related field
- 3+ years of experience in business analysis, product analytics, data analytics, AI operations, product operations, or related analytical roles
- Experience supporting digital products, customer-facing applications, conversational AI platforms, AI agents, chatbots, or generative AI solutions
- Experience defining KPIs, measuring product performance, building dashboards, and communicating insights to stakeholders
- Strong experience analyzing user behavior, customer journeys, operational metrics, usage data, and product performance indicators
- Working knowledge of Generative AI, LLMs, Conversational AI, Prompt-Based Systems, AI Agents, and Retrieval-Augmented Generation (RAG)
- Understanding of AI evaluation concepts including groundedness, relevance, factuality, hallucination risk, retrieval quality, response accuracy, and LLM evaluation methodologies
- Strong SQL skills and experience performing complex data analysis
- Experience with Power BI, Tableau, Adobe Analytics, or similar reporting and visualization tools
- Strong Excel skills including automation and advanced reporting functions
- Excellent analytical, communication, problem-solving, and stakeholder management skills
- Stable work history within large enterprise organizations
Desired Skills
- Experience with AI observability platforms such as LangSmith, Arize, or similar tools
- Experience with Python, Pandas, and advanced analytics techniques
- Familiarity with AWS services including DynamoDB and CloudWatch
- Experience with content metadata, taxonomies, and structured content management
- Experience supporting AI product operations, model evaluation, or AI governance initiatives
- Strong product analytics and experimentation experience
- Experience working within Agile product development environments
- 3+ years of Big 4 consulting experience preferred
