What is a language model benchmark?
A language model benchmark is a standardized set of tasks used to measure model performance under controlled conditions, producing comparable numbers across different models on identical hardware. Benchmarks answer specific questions: how often a model answers correctly, how fast it generates output, how much memory it consumes, and which capabilities it supports. No single benchmark captures a model completely. Accuracy on examinations, inference latency, memory footprint and functional support are distinct properties, and a model can be strong on one and weak on another. A useful evaluation measures several dimensions and reports them separately, which is the approach taken in this comparison of Amália 9B, Qwen 3.5-9B and Gemma 4-12B.
How were the three models compared?
The comparison used identical hardware and configuration for all three models, so any difference in results comes from the models themselves. Every model ran on AMD RX9700 32GB GPUs using the vLLM 0.24.0 inference engine, with a 16,384 token context window and 64 parallel requests, at native precision with no quantization. Qwen 3.5-9B ran with reasoning disabled. Five dimensions were examined: model architecture and memory use, per-token inference latency, agentic tool-calling support, accuracy on the PHEB examination benchmark, and informal Innergate AI · PHEB Benchmark Evaluation · Page 1 of 5 observations on Portuguese language and culture. The objective was to compare these three specific models under the same conditions, not to produce a general model ranking.
How do the models differ in architecture and memory use?
Model architecture determines the key-value cache footprint, which is the memory a model reserves per token during inference and the main driver of how many concurrent users a fixed GPU can serve. Amália 9B uses an older architecture, inherited from EuroLLM-9B: 42 layers, grouped-query attention with 8 key-value heads, no sliding window attention and no linear layers. This dated design produces a key-value cache roughly five times larger per token than Qwen 3.5-9B or Gemma 4-12B, which use more recent attention mechanisms. On the same GPU, Amália serves roughly five times fewer concurrent user sessions before exhausting memory. The disadvantage comes from the architecture, not from model size.
How do the models compare on inference speed?
Inference latency is measured as time per output token (TPOT), the time a model takes to generate each token, which determines throughput on fixed hardware. Measured at the 99th percentile, Amália 9B produced a TPOT of 46.67 milliseconds per token, against 32.81 milliseconds for Gemma 4-12B. Amália is about 42% slower per generated token than Gemma 4-12B at p99. Combined with the larger memory footprint, this means Amália serves fewer concurrent sessions and generates each response more slowly. For local deployment, the two properties compound and reduce the total request volume a single GPU can sustain.
Do the models support agentic tool calling?
Agentic tool calling is a model's built-in ability to invoke external functions, which is required for MCP integrations, agentic RAG and any pipeline that orchestrates tools. Support for it is defined in the model's chat template through dedicated tokens and a tools branch. Amália 9B has no tool-calling support in its chat template: no tools branch and no function-calling tokens. This is a training-time property, not a configuration option, so the capability cannot be added afterwards. Qwen 3.5-9B and Gemma 4-12B both ship with native tool calling and dedicated parsers. For workflows that depend on tool orchestration, Amália cannot be used.
How did the models perform on the PHEB benchmark?
The PHEB benchmark is a dataset of 1,819 Portuguese national secondary school examination questions across six disciplines, scored on exact correctness without partial credit. On overall accuracy, Gemma 4-12B scored 90.6%, Qwen 3.5-9B 89.6%, and Amália 9B 69.3%.
Model | Accuracy | Correct / Total | Total Time | Tokens per Response |
|---|---|---|---|---|
Gemma 4 12B | 90.6% | 1,648 / 1,819 | 44 min | ~0.5k |
Qwen 3.5 9B | 89.6% | 1,629 / 1,819 | 31 min | ~0.8k |
Amália 9B | 69.3% | 1,260 / 1,819 | 58 min | ~1.0k |
Gemma 4-12B and Qwen 3.5-9B finished within one point of each other, despite Qwen being smaller and running with reasoning disabled. Amália scored about 20 points lower, generated the most tokens per response, and took the longest to complete the set. On every metric in the table, Amália is the weakest of the three.
How did results vary by discipline?
Discipline-level results show where the accuracy gap is largest, detail that a single overall score hides. The chart below shows accuracy per discipline for all three models.
Mathematics: Gemma 4 97.3%, Qwen 3.5 92.5%, Amália 62.1%.
Philosophy: Gemma 4 92.5%, Qwen 3.5 90.1%, Amália 71.3%.
Geography: Qwen 3.5 89.6%, Gemma 4 89.2%, Amália 74.8%.
History: Gemma 4 88.3%, Qwen 3.5 88.3%, Amália 83.3%.
Biology and Geology: Gemma 4 87.3%, Qwen 3.5 87.3%, Amália 59.6%.
Portuguese Language: Gemma 4 87.6%, Qwen 3.5 85.0%,
Amália 70.3%. Amália scores below both other models in every discipline. The gap is smallest in History, where Amália still trails by about 5 points, and largest in Mathematics and Biology and Geology, where it falls roughly 25 to 35 points behind. Even in Portuguese Language, Amália scores about 15 points below both multilingual models.
How do the models handle Portuguese language and culture?
Language and cultural knowledge can be assessed informally before formal benchmarking, by prompting a model with region-specific facts and checking the responses. These observations are qualitative and not yet a scored benchmark, so they indicate direction rather than magnitude. In informal prompting, Amália 9B produced several confident errors on Portuguese subjects: it described the quindim as a traditional Portuguese almond-based sweet, when it is a Brazilian custard dessert; it attributed historical claims to Portuguese figures and places that are not supported by the sources; and it occasionally emitted German words in the middle of Portuguese responses. Qwen 3.5-9B and Gemma 4-12B did not produce these errors in the same prompts. A formal Portugueselanguage benchmark is required to quantify the difference.
What does the comparison show?
Across all five dimensions, Amália 9B is the weakest of the three models tested. Its architecture is older and reserves about five times more memory per token, its inference is about 42% slower at p99, it has no agentic tool-calling support, it scores about 20 points lower on the PHEB benchmark and below both other models in every discipline, and it produced factual errors on Portuguese subjects in informal testing. Amália is an early Portuguese-language model, and these numbers are a baseline against which future versions can be measured. The value of running the full comparison is that it replaces general claims with specific, reproducible numbers on architecture, inference, capabilities and accuracy.
Methodology and reproducibility
Evaluation dataset: PHEB, 1,819 questions across six disciplines, sourced from Portuguese national examination archives. Hardware: AMD RX9700 32GB GPUs. Inference engine: vLLM 0.24.0. Context window: 16,384 tokens. Concurrency: 64 parallel requests. Precision: native, no quantization. Qwen 3.5-9B tested with reasoning disabled. Architecture and tool-calling observations taken from published model weights and chat templates. Latency reported as p99 time per output token. Portuguese language observations are informal and not scored.