LLM Evaluation Assistant

We are seeking an LLM Evaluation Assistant to help our team verify the logic and accuracy of model generated financial insights. This role focuses on the careful review of automated outputs to ensure they meet our standards for factual discipline and economic reasoning.

About Blue Oak Consulting Blue Oak Consulting provides independent advisory on pricing, portfolio economics, capital allocation, and acquisition assumptions. We are not a traditional consulting firm focused on broad transformation programs or high level strategy decks. Instead, we interrogate specific commercial decisions where the financial stakes are high and the existing narrative is not sufficient.

Our team looks past corporate language and P&L complexity to identify where value is leaking or capital is trapped. We work across sectors like software, manufacturing, and healthcare to test whether the assumptions behind a growth plan or an acquisition actually hold under pressure. We are a small, principal led firm where the people who do the work are the ones accountable for the results.

We do not take transaction contingent fees, which allows us to be completely honest about whether the economics of a decision make sense. Our goal is to provide clarity through financial discipline and a healthy amount of skepticism. We stay through the execution phase because decisions that look sound in theory often fail when they meet real world friction.

The Role

As an LLM Evaluation Assistant, you will work in a fully remote capacity to test the outputs of large language models against real world financial data and logic. While many organizations are quick to adopt automated tools, we approach them with a disciplined eye, looking for errors, hallucinations, or flawed economic logic.

You will be part of a small team dedicated to finding the friction points where theory meets reality. This is a part time contract position designed for someone who enjoys investigation and has a high attention to detail.

What You Will Be Doing

Reviewing model outputs for factual accuracy regarding financial metrics and corporate filings. Testing the logical consistency of arguments presented by AI tools during the analysis of investment cases. Flagging instances where models use vague language instead of concrete data points. Comparing model generated summaries against source documents to identify omissions or distortions of risk factors.

Recording and categorizing errors in reasoning to help refine the firm's internal workflows. Assisting principals in verifying the math behind automated projections to ensure no basic arithmetic errors exist. Documenting the specific prompts used to generate outputs and noting which variations lead to more reliable results.

Collaborating with the team through digital channels to discuss cases where a model's interpretation of a contract is questionable.

What You Bring

• A current pursuit of or recent degree in finance, economics, or a related analytical field.
• A skeptical mindset that does not take automated or provided data at face value.
• Strong reading comprehension skills and the ability to spot subtle contradictions in a text.
• Basic familiarity with financial statements such as balance sheets and income statements.
• Comfort working in a fully remote setting where communication is largely written and focused on results.
• Proficiency in Excel and an interest in how data flows through different analytical systems.
• Willingness to perform repetitive tasks with a high degree of precision and focus.
• An organized approach to notes and documentation to help others review your findings easily.

What We Offer

A fully remote work schedule that provides flexibility for those with other commitments or studies. Attractive pay that is competitive for the role and reflecting the specialized nature of these evaluation tasks. Direct exposure to the work of senior principals who have extensive experience in corporate finance and advisory. A culture that values plain speaking, honesty, and analytical rigor over office politics.

Practical experience in the emerging field of AI evaluation within a commercial context. Access to complex, real world financial cases that go beyond what is typically taught in a classroom. An environment where your findings directly contribute to the quality of advice provided to clients. We believe that the best financial decisions are built on a foundation of verified facts rather than optimistic projections.

By joining our team as an assistant, you will play a part in maintaining the integrity of our analytical process. We value the quiet work of verification because it is what prevents costly errors in the long run. Working here means being part of a firm that prioritizes the truth of the numbers above all else.