From zero to eval in one afternoon
Fyntune integrates with your existing LLM pipeline without requiring infrastructure changes. Most teams are running automated evals on their first deploy within a few hours of signup.
Four steps to automated eval
Each step has a working code example. You don't need to change your existing LLM calls or move to a different model provider — just add the SDK wrapper around your existing call. Your production response path is unchanged.
Wrap your existing LLM calls with the Fyntune SDK. One decorator captures inputs, outputs, and metadata without changing your application logic. Supports Python and TypeScript. Webhook endpoint available for any other stack.
from fyntune import track
# Wrap your existing LLM call — no other changes
@track(feature="summarization", version="v12")
def summarize(text: str) -> str:
response = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": text}]
)
return response.choices[0].message.contentFyntune ships with 42 default eval criteria covering semantic similarity, factuality, guardrail compliance, response coherence, and tone consistency. Override thresholds per criterion, disable ones that don't apply to your use case, or add custom LLM-as-judge criteria in plain language via YAML. Criteria live in fyntune.yaml in your repo, versioned alongside your prompts and model configs.
# fyntune.yaml — lives in your repo
project: my-llm-app
features:
summarization:
eval_suite: default
thresholds:
semantic_similarity: 0.82
factuality: 0.90
guardrail_compliance: 0.98
custom_criteria:
- name: conciseness
max_tokens: 200Use the SDK or CLI to tag each prompt change as a new version. Fyntune stores the diff automatically and links eval results to specific prompt versions — so you always know which change introduced a regression.
# Tag a prompt version in your deploy script
fyntune prompt tag \
--feature summarization \
--version "v13" \
--file prompts/summarization.txt \
--message "Add citation requirement"Add the Fyntune step to your GitHub Actions, GitLab CI, or Vercel workflow. On each push, Fyntune runs your eval suite, compares against the previous version baseline, and returns a pass/fail verdict with per-criterion delta scores. Failures block the deploy and send a Slack or email alert.
# .github/workflows/deploy.yml
- name: Run Fyntune eval
uses: fyntune-ai/eval-action@v2
with:
api_key: ${{ secrets.FYNTUNE_API_KEY }}
block_on_regression: true
notify: slackZero-latency in your inference path
Fyntune runs async — it never sits in your user-facing request path. Eval runs happen in a separate worker pool triggered by your CI/CD hook.
Questions we hear before the first eval run
No. Fyntune runs asynchronously in a separate worker pool — it is never in your user-facing request path. The SDK wrapper captures call data and forwards it to the eval worker in the background. Your users experience zero added latency.
Fyntune supports OpenAI, Anthropic Claude, Mistral, LangChain, and LlamaIndex natively via SDK integrations. Any LLM accessible via REST API works through our webhook endpoint. We test new model releases from major providers within 48 hours of their public availability.
Yes. Fyntune's 42 default criteria can be adjusted by threshold or disabled entirely. You can also add custom criteria using natural language prompts (LLM-as-judge) or rule-based checks via YAML. Custom criteria live in your repo alongside your prompt files.
Eval data (LLM inputs and outputs captured for evaluation) is isolated per team and is not used to train any models. Raw LLM outputs are retained for a maximum of 30 days by default. Enterprise customers can configure custom retention windows and data residency options. See our Security page for full details.
Fyntune provides native GitHub Actions and GitLab CI integrations via official actions/plugins. Vercel deployments are supported via our deploy hook. Any CI system that can run a shell command can call the Fyntune CLI directly — the CLI outputs standard exit codes (0 = pass, 1 = regression detected).