About this role
Legal AI Evaluation Specialist
About the Role
We are seeking legal professionals to help evaluate the performance of advanced AI systems on real-world legal tasks. In this role, you will create realistic legal scenarios, assess AI-generated work products, and provide detailed feedback to improve the accuracy, reasoning, and quality of AI-generated legal analysis.
Key Responsibilities
- Develop realistic legal tasks based on professional experience in litigation, corporate law, compliance, intellectual property, employment law, real estate, or other legal practice areas.
- Create supporting materials such as contracts, pleadings, correspondence, policies, fact patterns, exhibits, and legal memoranda.
- Evaluate AI-generated deliverables, including legal analyses, contract redlines, research memos, brief sections, and issue-spotting exercises.
- Compare multiple AI responses and assess their accuracy, completeness, legal reasoning, and practical usefulness.
- Create detailed grading rubrics that define successful outcomes, required legal authorities, key issues, risk assessments, and appropriate recommendations.
- Identify and document errors, including missed legal issues, unsupported conclusions, fabricated authorities, incorrect legal standards, and contract provisions that were overlooked.
- Collaborate with other legal subject matter experts to review and improve evaluation tasks.
Qualifications
Required
- 2+ years of hands-on legal experience in a law firm, corporate legal department, government agency, compliance function, or related legal role.
- Strong legal research, analysis, and drafting skills.
- Excellent written communication with the ability to explain complex legal reasoning clearly and concisely.
- Professional or native-level English proficiency.
- Ability to critically evaluate legal work product and provide evidence-based feedback.
- Familiarity with AI tools such as ChatGPT, Claude, Gemini, or similar technologies.
- Strong attention to detail and ability to work independently.
Preferred
- Bachelor's degree (completed or in progress) required.
- JD, LLB, or equivalent law degree preferred.
- Experience in one or more of the following areas: Litigation (Civil, Commercial, Employment, or Administrative)
- Corporate & Transactional Law
- Mergers & Acquisitions
- Commercial Contract Review
- Compliance & Regulatory Affairs
- Intellectual Property
- Employment Law
- Real Estate Law
- Tax Law
- Legal Operations
- Bar admission is valued but not required.
- Experienced paralegals and legal analysts with substantial drafting responsibilities are encouraged to apply.
Ideal Candidate
- Currently works, or has recently worked, in a hands-on legal role.
- Can assess legal conclusions using statutes, regulations, case law, contractual language, and other primary sources.
- Has strong issue-spotting abilities and can identify inaccuracies, unsupported reasoning, and omitted risks.
- Demonstrates sound legal judgment and the ability to evaluate work across multiple practice areas.
- Is comfortable working with ambiguity and learning new evaluation frameworks quickly.
Additional Information
- Remote contract opportunity.
- Flexible schedule with a minimum availability of 10 hours per week.
- No maximum weekly hour requirement; full-time hours may be available.
- Candidates must be based in the United States, Canada, or the United Kingdom.
- Applicants with legal training from the U.S., U.K., Canada, Australia, or New Zealand are strongly encouraged to apply.
Compensation: Competitive hourly compensation based on legal experience and subject matter expertise. Schedule: Flexible, part-time to full-time availability.
#
Job Type & Location This is a Permanent position based out of New York, NY.
Pay and Benefits
The pay range for this position is $45000.00 - $60000.00/yr. Individual compensation offered for this position within this range will depend on many factors, including qualifications, skills, relevant experience, job knowledge, geographic location, internal equity, and other pertinent job-related factors. Independent contractor paid through paypal. No benefits offered
Job Type & Location
This is a Permanent position based out of New York, NY.
Application Deadline
This position is anticipated to close on Nov 24, 2026.
Workplace Type
This is a fully remote position.
About Aston Carter
Aston Carter provides world-class corporate talent solutions to thousands of clients across the globe. Specialized in accounting, finance, human resources, talent acquisition, procurement, supply chain and select administrative professions, we extend the capabilities of industry-leading companies. We draw on our deep recruiting expertise and expansive network to meet the evolving needs of our clients and talent community with agility and excellence. With offices across the U.S., Canada, Asia Pacific and Europe, Aston Carter serves many of the Fortune 500. We are proud to be a ClearlyRated Best of Staffing® Platinum Award winner for both client and talent service.
The company is an equal opportunity employer and will consider all applications without regard to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.
If you would like to request a reasonable accommodation, such as the modification or adjustment of the job application process or interviewing process due to a disability, please email [email protected] for other accommodation options.
San Francisco Fair Chance Ordinance: Pursuant to the San Francisco Fair Chance Ordinance, for all positions located in the city and county of San Francisco, we will consider for employment qualified applicants with arrest and conviction records.
Massachusetts Lie Detector: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.
Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our hiring process, including sourcing, screening, and evaluating candidates. AI helps assess applications and qualifications, but final decisions are made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools.