Coconut
AgentEvalsSpecialist
Neural analysis suggests this role is
optimal for Mid+ candidates.
“Agent Evals Specialist at Coconut. Skills: English Proficiency Rating, CEFR English proficiency level, written English, Consistent scoring, Detailed feedback. Read the source and the AI agent's output side by side.. Verify the content was captured accurately.”
What You'll Achieve.
improve the agent
Industry & Context.
40 hours/week, Preferred overlap anytime during 10AM to 10 PM PST
What They're Looking For.
Must Have
written English. Can read dense technical content for hours without losing focus, Consistent — your scoring on Monday matches your scoring on Friday, Clear, specific feedback: "section 4 dropped the key requirement from page 17", not "this is confusing"
Nice to Have
Prior work as an AI trainer, tutor, or evaluator (Outlier, DataAnnotation, xAI, Surge, Mercor, Invisible, Toloka, etc. ), Technical writing, editing, QA, translation, paralegal, or research-assistant background, Markdown familiarity
What You'll Do.
Read the source and the AI agent's output side by side., Verify the content was captured accurately., Review what the AI agent did.
What it created, changed, or left out., Score a short rubric covering accuracy, coverage, organization, and rule adherence., Write detailed feedback about the mistake., Submit.
Move to the next task.
How You'll Work.
Communication Scope
written English; Clear, specific feedback
Full Job Description
At**Coconut 🥥** , we connect founders and businesses with top-tier remote talent while prioritizing meaningful, reliable partnerships. We believe success happens when everyone wins. Clients achieve their goals, and our virtual professionals find purpose and growth in their work. We're currently looking for **top talent** to fill the role below: ** Role** : Agent Evals Specialist **Monthly Rate** : $900-1450 **Availability** : 40 hours/week **Time Zone** : Preferred overlap anytime during 10AM to 10 PM PST ================================================== To streamline your application, please carefully review and complete **ALL THREE** items below before proceeding to the application form. **What to Prepare:** 1️⃣ **English Proficiency Rating** – You will need to enter your CEFR English proficiency level in the application form. If you’re unsure of your rating, you can take a quick assessment here: [**EF SET Quick Check**](https://www.efset.org/quick-check/). 2️⃣ **Coconut VA Profile (Resume)** – Before submitting your application, please create your Coconut VA profile using this template: [**Coconut VA Profile Template**](https://bit.ly/CVAProfileTemplate). ⚠️⚠️⚠️ **YOUR RESUME MUST FOLLOW THIS FORMAT AND SHOULD BE IN ENGLISH.** **This is a test of how keen you are with details and how well you follow instructions. ⚠️⚠️⚠️** 3️⃣ **1-Minute Introduction Video** – Record a short introduction video (preferably using [**Loom**](https://www.loom.com/)) and provide the link in your application. Before you do, check out these [**video tips**](https://bit.ly/introvideotips) and [**sample videos**](https://bit.ly/IntroVidGuideAndSamples). **Important:** By submitting your video, you grant Coconut permission to share it on our platforms to help increase your chances of being matched with a client. Videos with broken or dummy links will result in disqualification. **Requirements** **Experience Needed** : 1\. Read the source and the AI agent's output side by side. Verify the
Applying for this Agent Evals Specialist role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
ANONYMOUS · UNFILTERED
What do employees actually say about Coconut?
Real rants from real employees. Read before you apply.