Custom Scorecards — Specmark Docs

Overview

Custom scorecards allow you to define repository-specific rules in addition to (or instead of) the default Specmark rules. Create a .specmark.yml file in the root of your repository to define custom checks.

YAML Schema

The .specmark.yml file must conform to the following top-level schema:

version: 1
rules:
  - id: unique-rule-id
    name: Human-readable rule name
    description: Optional detailed explanation
    category: documentation | testing | ci | ownership | security | maintenance
    weight: 1-10
    check:
      type: <check-type>
      # ... check-specific fields

Note: Rule id must be lowercase alphanumeric with hyphens only (e.g., has-readme).

Supported Check Types

1. file-exists

Verifies that a file exists at the specified path. Commonly used for documentation or configuration files.

- id: has-contributing-guide
  name: CONTRIBUTING.md exists
  category: documentation
  weight: 2
  check:
    type: file-exists
    path: CONTRIBUTING.md

Passes if: The file exists.
Fails if: The file does not exist.

2. file-not-exists

Verifies that a file does NOT exist. Useful for ensuring deprecated files have been removed.

- id: no-deprecated-config
  name: Old config.json removed
  category: maintenance
  weight: 1
  check:
    type: file-not-exists
    path: config.json

Passes if: The file does not exist.
Fails if: The file exists.

3. file-contains

Searches for a regex pattern within a file's content. Useful for verifying that documentation includes required sections or that config files contain specific keys.

- id: readme-has-badge
  name: README contains Specmark badge
  category: documentation
  weight: 1
  check:
    type: file-contains
    path: README.md
    pattern: 'specmark\.app/badge'

Passes if: The pattern is found in the file.
Fails if: The pattern is not found.
Skips if: The file does not exist or content is unavailable.

4. file-not-contains

Verifies that a pattern does NOT appear in a file. Useful for detecting forbidden strings like hardcoded secrets or legacy comments.

- id: no-hardcoded-api-key
  name: No hardcoded API keys in config
  category: security
  weight: 10
  check:
    type: file-not-contains
    path: config.yaml
    pattern: 'api[_-]?key:\s*["']?sk-'

Passes if: The pattern is not found.
Fails if: The pattern is found.
Skips if: The file does not exist.

5. file-min-lines

Ensures a file has a minimum number of non-empty lines. Useful for enforcing minimum documentation quality.

- id: substantial-readme
  name: README has at least 20 lines
  category: documentation
  weight: 3
  check:
    type: file-min-lines
    path: README.md
    minLines: 20

Passes if: The file has at least minLines non-empty lines.
Fails if: The file has fewer than minLines or does not exist.

6. dir-exists

Verifies that a top-level directory exists. Useful for enforcing project structure (e.g., docs/, tests/).

- id: has-tests-directory
  name: /tests directory exists
  category: testing
  weight: 5
  check:
    type: dir-exists
    path: tests

Passes if: The directory exists at the repository root.
Fails if: The directory does not exist.
Note: Only top-level directories are checked (no nested paths).

7. recent-activity

Checks that the repository has a minimum number of commits in the last 90 days. Useful for identifying stale or unmaintained repositories.

- id: active-maintenance
  name: At least 5 commits in last 90 days
  category: maintenance
  weight: 4
  check:
    type: recent-activity
    minCommits: 5
    days: 90

Passes if: The repository has at least minCommits in the last days.
Fails if: The repository has fewer commits or is archived.
Note: The days field is currently fixed at 90 and included for schema completeness.

Rule Categories

Each rule must specify one of the following categories:

documentation — README, CHANGELOG, API docs, etc.
testing — Test files, coverage, CI test runs
ci — CI/CD configuration, workflow files
ownership — CODEOWNERS, maintainer lists
security — Security policies, vulnerability scanning, secret detection
maintenance — Recent activity, dependency updates, deprecation cleanup

Rule Weights

Weights range from 1 (low priority) to 10 (critical). The overall score is the weighted average of passing rules. Higher-weight rules have a greater impact on the final grade.

Example weight guidelines:

1-2: Nice-to-haves (badges, changelog)
3-5: Standard best practices (tests, CI, documentation)
6-8: Important policies (CODEOWNERS, security scanning)
9-10: Critical requirements (no hardcoded secrets, license)

Full Example

Here's a complete .specmark.yml with multiple rule types:

version: 1
rules:
  # Documentation
  - id: has-readme
    name: README.md exists
    description: Repository must have a README with project overview
    category: documentation
    weight: 5
    check:
      type: file-exists
      path: README.md

  - id: readme-has-usage
    name: README includes usage section
    category: documentation
    weight: 3
    check:
      type: file-contains
      path: README.md
      pattern: '(?i)##\s*usage'

  - id: has-changelog
    name: CHANGELOG.md exists
    category: documentation
    weight: 2
    check:
      type: file-exists
      path: CHANGELOG.md

  # Testing
  - id: has-tests
    name: /tests directory exists
    category: testing
    weight: 7
    check:
      type: dir-exists
      path: tests

  # CI
  - id: has-github-actions
    name: GitHub Actions workflow configured
    category: ci
    weight: 6
    check:
      type: file-exists
      path: .github/workflows/ci.yml

  # Security
  - id: no-secrets-in-env
    name: No hardcoded secrets in .env.example
    category: security
    weight: 10
    check:
      type: file-not-contains
      path: .env.example
      pattern: 'sk-[a-zA-Z0-9]{32,}'

  - id: has-security-policy
    name: SECURITY.md exists
    category: security
    weight: 4
    check:
      type: file-exists
      path: SECURITY.md

  # Maintenance
  - id: actively-maintained
    name: At least 3 commits in last 90 days
    category: maintenance
    weight: 5
    check:
      type: recent-activity
      minCommits: 3
      days: 90

  # Ownership
  - id: has-codeowners
    name: CODEOWNERS file exists
    category: ownership
    weight: 6
    check:
      type: file-exists
      path: .github/CODEOWNERS

Validation and Errors

Specmark validates your .specmark.yml using Zod. If validation fails, you'll see an error in the scan log with details about the invalid field. Common errors:

Rule id must be lowercase alphanumeric with hyphens only — Use has-readme, not hasReadme
Weight must be between 1 and 10 — Adjust the weight to be within range
Invalid check type — Ensure the type matches one of the supported check types above

Merge Behavior

Custom rules are merged with the default Specmark scorecard. If a custom rule has the same id as a default rule, the custom rule overrides it. To disable a default rule entirely, you cannot currently do so — this is on the roadmap for a future release.