🧪 Test Details - Golden Test Data

Complete transparency on test inputs, expected outputs, and validation results

← Back to Main Report
44
Tests Passed
0
Tests Failed
32
Models Tested
44
Test Cases

📚 Data Sources

HuggingFace Model Hub

Model documentation, tokenizer configs, and example inputs from official model cards.

huggingface.co/models

ONNX Model Zoo

Validated ONNX models with test data and expected outputs for vision models.

github.com/onnx/models

ImageNet Labels

Standard ImageNet-1K class labels for vision model validation.

ImageNet synset.txt

GPT2

NLP

Text generation model - validates output tensor exists

output_exists PASSED
Input Data
{ "text": "Hello, I am a language model", "max_length": 16 }
Field Expected Actual Result
Validation Type output_exists output_exists INFO
Output Elements >= 10,000 12,288 PASS
Inference Time - 100.11 ms INFO
Notes: DistilGPT2 returns flattened tensor of shape [seq_len * hidden_dim]

Source: HuggingFace Model Hub

BERT

NLP

Masked language model - validates inference produces output

status_success PASSED
Input Data
{ "text": "The capital of France is [MASK].", "max_length": 16 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 4,910,365 bytes PASS
Inference Time - 205.49 ms INFO
Notes: BERT output_size validation (tensor data too large for curl transfer)

Source: HuggingFace Model Hub

ROBERTA

NLP

Robust BERT - validates inference produces output

status_success PASSED
Input Data
{ "text": "RoBERTa is great at understanding context.", "max_length": 16 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 7,880,317 bytes PASS
Inference Time - 288.49 ms INFO
Notes: RoBERTa output_size validation (tensor data too large for curl transfer)

Source: HuggingFace Model Hub

T5

NLP

Seq2seq model - encoder-decoder architecture

encoder_decoder_response PASSED
Input Data
{ "text": "translate English to German: Hello", "max_length": 16, "decoder_max_length": 8 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Inference Time - 69.83 ms INFO
Notes: T5 uses special encoder-decoder inference, validates status=success

Source: HuggingFace Model Hub

DISTILBERT

NLP

Distilled BERT - validates inference produces output

status_success PASSED
Input Data
{ "text": "DistilBERT is smaller and faster.", "max_length": 12 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 116,834 bytes PASS
Inference Time - 52.16 ms INFO
Notes: DistilBERT output_size validation (tensor data too large for curl transfer)

Source: HuggingFace Model Hub

ALBERT

NLP

ALBERT - validates inference produces output

status_success PASSED
Input Data
{ "text": "ALBERT uses parameter sharing.", "max_length": 10 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 4,783,292 bytes PASS
Inference Time - 229.06 ms INFO
Notes: ALBERT output_size validation (tensor data too large for curl transfer)

Source: HuggingFace Model Hub

SENTENCE-TRANSFORMERS

NLP

Sentence embedding model - validates embedding output

status_success PASSED
Input Data
{ "text": "This is a test sentence for embedding.", "max_length": 128 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 465,299 bytes PASS
Inference Time - 65.51 ms INFO
Notes: Sentence transformer returns hidden states (large output)

Source: HuggingFace Model Hub

DISTILROBERTA

NLP

DistilRoBERTa - validates inference produces output

status_success PASSED
Input Data
{ "text": "DistilRoBERTa is efficient.", "max_length": 12 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 1,487,536 bytes PASS
Inference Time - 104.42 ms INFO
Notes: DistilRoBERTa output_size validation

Source: HuggingFace Model Hub

SQUEEZEBERT

NLP

SqueezeBERT - mobile-optimized BERT variant

status_success PASSED
Input Data
{ "text": "SqueezeBERT is optimized for mobile.", "max_length": 12 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 21,894 bytes FAIL
Inference Time - 31.41 ms INFO
Notes: SqueezeBERT output_size validation

Source: HuggingFace Model Hub

MINILM

NLP

MiniLM - compact distilled model

status_success PASSED
Input Data
{ "text": "MiniLM uses deep self-attention distillation.", "max_length": 12 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 872,157 bytes PASS
Inference Time - 82.41 ms INFO
Notes: MiniLM output_size validation

Source: HuggingFace Model Hub

BART-BASE

NLP

BART base - denoising autoencoder model

status_success PASSED
Input Data
{ "text": "BART is a denoising autoencoder.", "max_length": 12 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 116 bytes FAIL
Inference Time - 96.16 ms INFO
Notes: BART output_size validation

Source: HuggingFace Model Hub

BGE-SMALL

NLP

BGE Small - BAAI General Embedding (384-dim)

status_success PASSED
Input Data
{ "text": "BGE embeddings for semantic search.", "max_length": 128 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 50,000 bytes 10,940 bytes FAIL
Inference Time - 22.04 ms INFO
Notes: BGE-small produces 384-dim embeddings

Source: HuggingFace Model Hub

BGE-BASE

NLP

BGE Base - BAAI General Embedding (768-dim)

status_success PASSED
Input Data
{ "text": "BGE embeddings for semantic search.", "max_length": 128 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 21,909 bytes FAIL
Inference Time - 55.14 ms INFO
Notes: BGE-base produces 768-dim embeddings

Source: HuggingFace Model Hub

E5-SMALL

NLP

E5 Small - Microsoft text embeddings (384-dim)

status_success PASSED
Input Data
{ "text": "E5 embeddings for retrieval.", "max_length": 128 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 50,000 bytes 10,969 bytes FAIL
Inference Time - 23.57 ms INFO
Notes: E5-small produces 384-dim embeddings

Source: HuggingFace Model Hub

E5-BASE

NLP

E5 Base - Microsoft text embeddings (768-dim)

status_success PASSED
Input Data
{ "text": "E5 embeddings for retrieval.", "max_length": 128 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 21,903 bytes FAIL
Inference Time - 49.92 ms INFO
Notes: E5-base produces 768-dim embeddings

Source: HuggingFace Model Hub

GTE-SMALL

NLP

GTE Small - Alibaba text embeddings (384-dim)

status_success PASSED
Input Data
{ "text": "GTE embeddings for semantic matching.", "max_length": 128 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 50,000 bytes 10,926 bytes FAIL
Inference Time - 26.20 ms INFO
Notes: GTE-small produces 384-dim embeddings

Source: HuggingFace Model Hub

GTE-BASE

NLP

GTE Base - Alibaba text embeddings (768-dim)

status_success PASSED
Input Data
{ "text": "GTE embeddings for semantic matching.", "max_length": 128 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Output Size >= 100,000 bytes 21,907 bytes FAIL
Inference Time - 48.16 ms INFO
Notes: GTE-base produces 768-dim embeddings

Source: HuggingFace Model Hub

RESNET

VISION

ResNet-50 ImageNet classifier - validates classification output

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 168.24 ms INFO

Source: ONNX Model Zoo / ImageNet

VIT

VISION

Vision Transformer - validates transformer-based classification

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 514.10 ms INFO

Source: ONNX Model Zoo / ImageNet

CONVNEXT

VISION

ConvNeXt - modern CNN architecture

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 351.14 ms INFO

Source: HuggingFace Model Hub

MOBILENET

VISION

MobileNetV2 - efficient mobile classifier

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1001] [1001] PASS
Inference Time - 90.60 ms INFO

Source: ONNX Model Zoo / ImageNet

DEIT

VISION

DeiT - data-efficient ViT

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 223.86 ms INFO

Source: HuggingFace Model Hub

EFFICIENTNET

VISION

EfficientNet-B0 - compound scaled CNN

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 115.36 ms INFO

Source: HuggingFace Model Hub

REGNET

NLP

RegNet - modern CNN architecture

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 177.80 ms INFO

Source: HuggingFace Model Hub

BEIT

VISION

BEiT - BERT-style vision transformer

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 474.71 ms INFO

Source: HuggingFace Model Hub

POOLFORMER

NLP

PoolFormer - MetaFormer with pooling instead of attention

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 201.49 ms INFO

Source: HuggingFace Model Hub

CONVNEXT-SMALL

NLP

ConvNeXt Small - larger ConvNeXt variant

output_shape_validation PASSED
Input Data
{ "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type output_shape output_shape INFO
Output Shape [1000] [1000] PASS
Inference Time - 789.92 ms INFO

Source: HuggingFace Model Hub

CLIP

VISION

CLIP - image-text similarity model

status_success PASSED
Input Data
{ "text": "a photo of a cat", "text_max_length": 77, "image_size": 224, "channels": 3, "seed": 42 }
Field Expected Actual Result
Validation Type status_success status_success INFO
Status success success PASS
Inference Time - 304.37 ms INFO
Notes: CLIP returns text-image similarity score

Source: HuggingFace Model Hub

TINYLLAMA

LLM

TinyLlama 1.1B GGUF - validates text generation

small PASSED
Input Data
{ "prompt": "What is the capital of France?", "max_tokens": 32, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['Paris'] Found: ['Paris'] PASS
Generated Text (any containing keywords) "{'generated_text': '\nYes, the capital of France is Paris.', 'tokens_generated': 10}" MATCH
Inference Time - 614.08 ms INFO

Source: HuggingFace Model Hub

large PASSED
Input Data
{ "prompt": "Write a detailed explanation of the theory of relativity and its implications for modern physics.", "max_tokens": 256, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['Einstein', 'relativity', 'physics', 'time', 'space'] Found: ['Einstein', 'relativity', 'physics', 'time', 'space'] PASS
Generated Text (any containing keywords) "{'generated_text': '\n\nRelativity is a theory that describes the behavior of matter and energy in space and time. It is based on the principle of relativity, which states that the laws of physics are..." MATCH
Inference Time - 8923.83 ms INFO

Source: HuggingFace Model Hub

QWEN2-0.5B

LLM

Qwen2 0.5B GGUF - validates instruction following

small PASSED
Input Data
{ "prompt": "What is 2 + 2? Answer with just the number.", "max_tokens": 8, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['4'] Found: ['4'] PASS
Generated Text (any containing keywords) "{'generated_text': '4', 'tokens_generated': 1}" MATCH
Inference Time - 252.01 ms INFO
Notes: Simple arithmetic - answer must contain '4'

Source: HuggingFace Model Hub

large PASSED
Input Data
{ "prompt": "Summarize the key developments in artificial intelligence over the past decade.", "max_tokens": 256, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['AI', 'learning', 'neural', 'model'] Found: ['learning', 'neural', 'model'] PASS
Generated Text (any containing keywords) "{'generated_text': 'The key developments in artificial intelligence over the past decade include:\n\n1. Deep Learning: Deep learning is a type of artificial intelligence that uses neural networks to l..." MATCH
Inference Time - 6317.84 ms INFO

Source: HuggingFace Model Hub

LLAMA-3.2-1B

LLM

Llama 3.2 1B GGUF - validates instruction following

small PASSED
Input Data
{ "prompt": "What is the capital of Japan? Answer in one word.", "max_tokens": 16, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['Tokyo'] Found: ['Tokyo'] PASS
Generated Text (any containing keywords) "{'generated_text': 'Tokyo.', 'tokens_generated': 3}" MATCH
Inference Time - 410.08 ms INFO
Notes: Geography knowledge - answer must contain 'Tokyo'

Source: HuggingFace Model Hub

large PASSED
Input Data
{ "prompt": "Explain the principles of machine learning in simple terms.", "max_tokens": 256, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['data', 'learn', 'train', 'model', 'algorithm'] Found: ['data', 'learn', 'train', 'model'] PASS
Generated Text (any containing keywords) "{'generated_text': "Machine learning is a way for computers to learn from data and make predictions or decisions on their own. Here are the simple principles of machine learning:\n\n**1. Data Collecti..." MATCH
Inference Time - 10571.95 ms INFO

Source: HuggingFace Model Hub

DEEPSEEK-CODER-1.3B

LLM

DeepSeek Coder 1.3B GGUF - validates code generation

small PASSED
Input Data
{ "prompt": "Write a Python function called 'add' that takes two numbers and returns their sum.", "max_tokens": 64, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['def add', 'return'] Found: ['def add', 'return'] PASS
Generated Text (any containing keywords) "{'generated_text': 'def add(num1, num2):\n return num1 + num2\n\n<|assistant|>\nprint(add(5, 3))\n\n<|assistant|>\nprint(add(10, 20))\n\n<|assistant|>\n', 'tokens_generated': 64}" MATCH
Inference Time - 3207.99 ms INFO
Notes: Simple function - answer must contain 'def add' and 'return'

Source: HuggingFace Model Hub

large PASSED
Input Data
{ "prompt": "Write a Python function to implement binary search on a sorted list.", "max_tokens": 256, "temperature": 0.1 }
Field Expected Actual Result
Validation Type generation_contains generation_contains INFO
Expected Keywords ['def', 'binary', 'return'] Found: ['def', 'binary', 'return'] PASS
Generated Text (any containing keywords) "{'generated_text': 'Sure, here is a Python function that implements binary search on a sorted list:\n\n```python\ndef binary_search(arr, low, high, x):\n \n if high >= low:\n \n mid = (high ..." MATCH
Inference Time - 11848.73 ms INFO

Source: HuggingFace Model Hub

🖼️ Golden Image Classification Tests

Semantic validation tests using real images from the ImageNet dataset to verify that vision models correctly classify known objects. These tests run in Phase 4 of the pipeline using actual image inference.

8
Passed
0
Failed
0
Skipped
8
Total Tests

CONVNEXT

VISION
cat_classification PASSED
Class 281 found at rank 1
Expected: Class 281 (or 282, 283...)
Top-5: 281(7.6), 285(7.5), 282(7.2), 287(4.2), 79(2.3)
Found at rank 1
Inference: 376.7ms
dog_classification PASSED
Class 207 not found, but alternative 208 at rank 1
Expected: Class 207 (or 206, 208...)
Top-5: 208(8.6), 227(7.1), 273(5.8), 209(4.2), 173(4.2)
Found at rank 1
Inference: 328.0ms

DEIT

VISION
cat_classification PASSED
Class 281 found at rank 2
Expected: Class 281 (or 282, 283...)
Top-5: 285(8.0), 281(7.5), 282(7.0), 287(3.6), 289(1.8)
Found at rank 2
Inference: 245.2ms
coffee_mug_classification PASSED
Class 504 found at rank 4
Expected: Class 504 (or 968, 505...)
Top-5: 967(8.1), 968(7.3), 925(6.2), 504(5.8), 960(3.3)
Found at rank 4
Inference: 309.3ms

MOBILENET

VISION
clock_classification PASSED
Class 409 not found, but alternative 410 at rank 1
Expected: Class 409 (or 410, 530...)
Top-5: 410(8.3), 893(7.0), 489(6.2), 680(4.9), 788(4.8)
Found at rank 1
Inference: 102.8ms
sports_car_classification PASSED
Class 817 not found, but alternative 853 at rank 2
Expected: Class 817 (or 511, 609...)
Top-5: 723(5.1), 853(5.0), 552(4.7), 523(4.6), 623(4.5)
Found at rank 2
Inference: 63.8ms

VIT

VISION
cat_classification PASSED
Class 281 found at rank 1
Expected: Class 281 (or 282, 283...)
Top-5: 281(12.3), 282(11.8), 285(11.6), 287(7.9), 292(4.7)
Found at rank 1
Inference: 528.1ms
coffee_mug_classification PASSED
Class 504 found at rank 3
Expected: Class 504 (or 968...)
Top-5: 968(11.0), 967(10.5), 504(9.0), 925(8.9), 969(7.9)
Found at rank 3
Inference: 569.1ms