Quality Analyst II

About us HighLevel is an AI-powered business operating system that gives agencies, entrepreneurs and SMBs the infrastructure to build, automate and scale. Today, HighLevel supports SMBs across 150+ countries, fueling community-driven growth rooted in real customer outcomes. To date, businesses operating on HighLevel have generated over $7 billion in ecosystem value, demonstrating the impact of shared infrastructure at scale. By centralizing conversations, automation and intelligence into one system, we help businesses move faster, reduce complexity and execute efficiently. Behind the platform, HighLevel powers more than 4 billion API hits and 2.5 billion message events daily. With 250 terabytes of distributed data, 250+ microservices and over 1 million domain names supported, our architecture is built for performance, resilience and long-term scalability. Our people With over 2,000 team members across 10+ countries, HighLevel operates as a global, remote-first organization built for speed and ownership. We value initiative, clarity and execution, creating space for ambitious people to build systems that support millions of businesses worldwide. Here, innovation thrives, ideas are celebrated and people come first, no matter where they call home. Our impact Every month, HighLevel enables more than 1.5 billion messages, 200 million leads and 20 million conversations for the more than 1 million businesses we support. Behind those numbers are real people building independence, expanding opportunity and creating measurable impact. We’re proud to be a part of that. Learn more about us on our YouTube Channel or Blog Posts  Role Overview We are looking for a Quality Analyst II to join Customer Success — Programs & AI Innovation as an evaluator-first specialist focused on the quality of AI-powered workflows that support our advisors before, during and after every customer call. You will spend most of your time evaluating AI output across ORA, our internal AI platform — reviewing pre-call briefs, in-call guidance, post-call grading and transcription-driven customer follow-ups. You will classify failure modes, document patterns and feed structured findings into the team that builds inside ORA. You will also do light hands-on work inside ORA — refining prompts, updating knowledge base entries and tweaking existing skills — closing the loop from “found an issue” to “shipped a fix.” This is a role with a deliberate growth path. You start in an evaluator-first lane with light creation work. As you build judgment, exceed expectations and demonstrate strong ownership, your scope expands toward deeper creation work alongside the Lead Analyst on the team. Key Responsibilities 1. Evaluate AI Quality Across the Advisor Stack You are the quality owner for advisor-facing AI inside ORA. You review AI output across the full call lifecycle and surface what is working, what is broken and where the bar needs to be raised. Review pre-call briefs for accuracy, completeness and signal quality — customer context, adoption signals, retention flags Evaluate in-call guidance — next-best-action prompts, suggestions and recommendation flows — for relevance and accuracy in live conversation Score post-call outputs: quality grading, AI-generated summaries, CRM updates, sentiment signals Audit transcription-driven follow-ups — customer updates, recap notes, follow-up tasks — for accuracy, tone and customer impact Classify issues by type and severity — hallucination, missing context, wrong action, tone mismatch, KB gap — using standardized rubrics Participate in calibration sessions to keep scoring consistent across reviewers 2. Iterate Inside ORA — Light Creation Work You close the loop. When you spot an issue you can fix without engineering involvement, you fix it inside ORA. Refine and iterate on prompts within existing skills and sub-agents Update knowledge base entries to close information gaps surfaced in reviews Tweak existing skills and orchestrations to improve output quality Test changes, validate improvements and roll out updates inside ORA Document what you changed and why — every iteration is a data point 3. Surface Trends & Drive Continuous Improvement You connect individual evaluations to systemic insights. Leadership and the Lead Analyst rely on you to know where the next investment should land. Tag conversations by product area, severity and issue type with consistent classification Maintain a clear, trusted log of issue types, trends and conversation behaviors Surface recurring patterns and systemic gaps to leadership and the Lead Analyst Recommend prioritized improvement opportunities — what to fix, what to build, what to redesign Build reporting that gives Success leadership visibility into AI quality and the improvement backlog 4. Partner with RevOps Engineering & the Lead Analyst When a fix requires ORA platform changes or end-to-end design, you stay close. Bring structured findings, repro examples and quality criteria when a platform change is needed Partner with the Lead Analyst on end-to-end builds — bring evaluation signal, test the output, validate the bar Co-define quality criteria for new sub-agents, skills and orchestrations before they ship Stay close through the build to ensure improvements land as intended 5. Stay Sharp: Learn, Experiment, Share The AI space moves weekly. We hire people who keep up on their own and bring what they learn back. Stay current on LLM behavior, prompt patterns and AI evaluation techniques Run small experiments with new prompts, tools or evaluation approaches inside ORA Share what you learn with the broader Success team and the Lead Analyst Requirements Insatiable curiosity and a self-learning mindset — this is non-negotiable. The AI space changes weekly and we hire people who level up on their own without waiting for permission or perfect instructions. 2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role where you reviewed and improved customer-facing or advisor-facing work Working understanding of how LLMs behave — prompt design, common failure modes, conversational AI patterns General understanding of databases and how data flows between systems Comfort reading code and making minor edits — enough to follow integrations, debug prompt behavior, iterate inside ORA and hold informed technical conversations with engineering. You do not need to be a full-stack engineer. Strong written communication and attention to detail — you can document a failure mode so clearly that an engineer can act on it without follow-up Ability to follow structured guidelines and scoring rubrics consistently while suggesting improvements to them Comfort working with evolving tools and AI-assisted workflows Self-starter mentality — you can manage review volume, prioritize what matters and move work forward with minimal direction Bachelor’s degree in a related field, or equivalent practical experience Preferred Qualifications Hands-on experience with AI development platforms such as Claude, Cursor or similar tools Familiarity with retrieval-augmented generation (RAG), vector databases or knowledge-base architecture Familiarity with SQL or similar database querying Experience with APIs, webhooks or data integration between systems Background in customer success, support, sales enablement or call center environments Experience with the HighLevel platform or comparable SaaS products Prior experience evaluating AI-generated conversations, chatbots or automated support output