How Our AI Model Recommendations Work

Selecting an AI model is no longer a question of capability alone, but of fit. With an ever-expanding ecosystem of models varying in cost, performance, latency, and specialization, choosing the right one has become a multi-dimensional decision problem rather than a simple comparison exercise. That is exactly the problem the AI Model Recommendation Wizard is designed to solve.

The wizard evaluates your specific requirements and systematically scores models from a predefined catalog to identify the most suitable options. Instead of relying on generic benchmarks or one-size-fits-all rankings, it customizes its recommendations based on the context in which the model will be used.

This document provides a detailed explanation of how the scoring mechanism works, the assumptions built into the system, and the limitations you should be aware of when interpreting its recommendations.

The five-step process

1. You select your industry (Software, Healthcare, Finance, etc.)
2. You pick a use case (Code Generation, Chatbot, Fraud Detection, etc.) or describe a custom one
3. You select your scale (Hobby, Startup, Growing, Enterprise)
4. You choose one or two priorities (Cost, Quality, Speed, Privacy, Integration)
5. The system scores and ranks all matching models, returning the top 5

For background on what factors matter when picking a model, see How to Choose the Right AI Model.

How scoring works: the 70/25/5 model

Each model receives a score from 0 to 100, composed of three weighted signals:

Use case match (70% of score)

This is the primary signal, combining three sub-scores.

Tag matching (up to 50 points)

Every model in the catalog has use case tags (e.g., "code-generation", "chatbot", "finance"). These are matched against your selection:

• Exact match (model tagged with your use case): 50 points
• Related use case in the same domain: 40 points
• General domain match: 30 points
• General-purpose model: 20 points
• Specialist in a different domain: 5 points

For custom use cases, the system extracts keywords from your description and matches them against model tags and descriptions. This produces meaningful scores even for use cases not in the predefined list, though precision is lower than for predefined matches.

Capability matching (up to 35 points)

Each use case has preferred capabilities (e.g., Code Generation prefers code_generation, function_calling, streaming). The score reflects what percentage of these the model supports:

Points = 35 × (capabilities matched / capabilities required)

Context window fit (up to 15 points)

Each use case has a minimum context window requirement. Models are scored based on how much they exceed it:

• 4x or more the requirement: 15 points
• 2x the requirement: 12 points
• Meets the requirement: 8 points
• Below but usable (50%+): 4 points

Priority score (25% of score)

Based on your selected priority, a specific scoring method applies.

Cost priority:

• Blended cost = (input cost + output cost x 2) / 3 per million tokens
• Under $0.50: 95 points | $0.50-2: 80 | $2-5: 65 | $5-10: 50 | $10-20: 30 | Over $20: 15
• Scale multiplier: Hobby projects weight cost at 1.5x, enterprise at 0.75x

Quality priority:

• Averaged benchmark scores across relevant benchmarks for the use case
• Code use cases check HumanEval, MBPP, LiveCodeBench
• Reasoning use cases check MMLU, GPQA, ARC
• General use cases check a broad mix

Speed priority:

• Based on tokens-per-second throughput
• Over 150 tok/s: 95 points | 100-150: 80 | 50-100: 65 | 20-50: 40 | Under 20: 25

Privacy priority:

• Open source (weights downloadable): +40 points
• Permissive license (Apache, MIT, Llama): +15 points
• Self-hosting deployment options: +15 points
• Small enough for single-GPU hosting (under 13B): +10 points

Integration priority:

• Function calling / tool use: +25 points
• JSON mode / structured output: +15 points
• Streaming support: +10 points
• Documentation available: +10 points

Popularity (5% of score)

Featured or verified models receive a small boost. This prevents obscure models from ranking above well-tested alternatives that have broader adoption.

How candidate models are filtered

Before scoring, the catalog is filtered to find relevant candidates:

1. Tag overlap: Models must have at least one tag matching your industry domain or use case
2. Required capabilities: If your use case needs specific capabilities (e.g., vision for quality control), models without them are excluded
3. Modality requirements: If you need image or audio input, models without those modalities are excluded
4. Candidate limit: The system evaluates up to 200 matching models

Open-source inclusion guarantee

When showing "All Models" results, the system guarantees at least one open-source model appears in the top 5. If all top 5 are proprietary, the lowest-ranked one is swapped with the highest-scoring open-source alternative. You can also toggle "Open Source Only" to see exclusively open-source recommendations.

Auto-tagging

Models in the catalog are automatically tagged using rule-based analysis of their:

• Name patterns: "Coder" maps to software, "Med" maps to healthcare
• Capabilities: function_calling maps to api-integration, code_generation maps to software
• Benchmarks: HumanEval scores indicate code specialization
• Model family: GPT-4, Llama, Gemini map to general purpose

Tags can be overridden manually for accuracy. The auto-tagger runs when new models are added to the catalog.

What affects recommendation quality

The quality of recommendations depends on data coverage:

• Use case tags: how thoroughly each model's strengths have been tagged
• Capabilities: whether the model's feature set (function calling, vision, etc.) is recorded
• Benchmarks: performance data for quality-based scoring
• Pricing: cost data for cost-based scoring
• Context window: for context-sensitive use cases (RAG, document analysis)

Data coverage is improved through automated enrichment pipelines and manual review, but gaps remain. Models added recently or from smaller providers tend to have less complete data.

Known limitations

1. Scoring is rule-based. The system uses predefined weights and thresholds, not ML-based relevance. Edge cases may not score optimally. For example, a model that excels at a niche task but is not tagged for it will score lower than it should.

2. Custom use cases get less precise scoring. When you type a custom use case, the system extracts keywords and matches against model descriptions. This works for broad categories but is less accurate than predefined use case scoring, where each use case has hand-tuned capability requirements.

3. Data coverage varies. Not all models have complete benchmark, pricing, or capability data. Models with missing data score conservatively (middle of the range), which means they may rank lower than they deserve.

4. No real-time performance data. Speed and throughput scores are based on published specs, not live benchmarks. Actual performance depends on provider load, quantization settings, and hardware configuration.

5. Priority weights are fixed. The 70/25/5 split works well for most queries, but some users may care about cost and quality equally. The current system does not allow custom weight adjustment.

---

Try the AI Model Recommendation Wizard for your use case, or browse models directly in the model catalog. For a broader framework on model selection, see How to Choose the Right AI Model.