benchflow-ai

skillsbench

@benchflow-ai/skillsbench
benchflow-ai
230
165 forks
Updated 1/18/2026
View on GitHub

SkillsBench contribution workflow. Use when: (1) Creating benchmark tasks, (2) Understanding repo structure, (3) Preparing PRs for task submission.

Installation

$skills install @benchflow-ai/skillsbench
Claude Code
Cursor
Copilot
Codex
Antigravity

Details

Repositorybenchflow-ai/skillsbench
Path.claude/skills/skillsbench/SKILL.md
Branchmain
Scoped Name@benchflow-ai/skillsbench

Usage

After installing, this skill will be available to your AI coding assistant.

Verify installation:

skills list

Skill Instructions

name: skillsbench description: "SkillsBench contribution workflow. Use when: (1) Creating benchmark tasks, (2) Understanding repo structure, (3) Preparing PRs for task submission."

SkillsBench

Benchmark evaluating how well AI agents use skills.

Official Resources

Repo Structure

tasks/<task-id>/          # Benchmark tasks
.claude/skills/           # Contributor skills (harbor, skill-creator)
CONTRIBUTING.md           # Full contribution guide

Local Workspace & API Keys

  • .local-workspace/ - Git-ignored directory for cloning PRs, temporary files, external repos, etc.
  • .local-workspace/.env - May contain ANTHROPIC_API_KEY and other API credentials. Check and use when running harbor with API access.

Quick Workflow

# 1. Install harbor and initialize a task
uv tool install harbor
harbor tasks init tasks/<task-name>

# 2. Write files (see harbor skill for templates)

# 3. Validate
harbor tasks check tasks/my-task
harbor run -p tasks/my-task -a oracle  # Must pass 100%

# 4. Test with agent
# you can test agents with your claude code/cursor/codex/vscode/GitHub copilot subscriptions, etc.
# you can also trigger harbor run with your own API keys.
# for PR we only need some idea of how sota level models and agents perform on the task with and without skills.
harbor run -p tasks/my-task -a claude-code -m 'anthropic/claude-opus-4-5'

# 5. Submit PR
# please see our PR template for more details.

Task Requirements

  • Things people actually do
  • Agent should be more performant with skills than without skills.
  • task.toml and solve.sh must be human-authored
  • Prioritize verifiable tasks. LLM-as-judge or agent-as-a-judge is not priority right now. But we are open to these ideas and will brainstorm if we can make these tasks easier to verify.

References

More by benchflow-ai

View all
fjsp-baseline-repair-with-downtime-and-policy
230

Repair an (often imperfect) Flexible Job Shop Scheduling baseline into a downtime-feasible, precedence-correct schedule while staying within policy budgets and matching the evaluator’s exact metrics and “local minimal right-shift” checks.

temporal-python-testing
230

Test Temporal workflows with pytest, time-skipping, and mocking strategies. Covers unit testing, integration testing, replay testing, and local development setup. Use when implementing Temporal workflow tests or debugging test failures.

locational-marginal-prices
230

Extract locational marginal prices (LMPs) from DC-OPF solutions using dual values. Use when computing nodal electricity prices, reserve clearing prices, or performing price impact analysis.

Managed Package Architecture
230

This skill should be used when the user asks to "design package structure", "create managed package", "configure 2GP", "set up namespace", "version management", or mentions managed package topics like "LMA", "subscriber orgs", or "package versioning". Provides comprehensive guidance for second-generation managed package (2GP) architecture, ISV development patterns, and package lifecycle management.