Testing

Structyl provides a language-agnostic reference test system. Tests are written in JSON and run against all language implementations.

Why Reference Tests?

When you maintain the same library in multiple languages, you need to ensure they all behave identically. Reference tests solve this by:

• Defining test cases in JSON (language-neutral)
• Running the same tests against all implementations
• Comparing outputs with configurable tolerance

Test Structure

Basic Test File

Tests live in the tests/ directory. Each JSON file is a test case:

{
  "input": {
    "x": [1.0, 2.0, 3.0, 4.0, 5.0]
  },
  "output": 3.0
}

Directory Layout

tests/
├── mean/                    # Test suite: "mean"
│   ├── basic.json          # Test case
│   ├── empty.json
│   └── negative.json
├── variance/                # Test suite: "variance"
│   └── ...
└── regression/              # Test suite: "regression"
    └── ...

Input and Output Types

Scalar Values

{
  "input": { "value": 42 },
  "output": 84
}

Arrays

{
  "input": {
    "x": [1.0, 2.0, 3.0]
  },
  "output": [2.0, 4.0, 6.0]
}

Objects

{
  "input": {
    "data": [1, 2, 3, 4, 5]
  },
  "output": {
    "mean": 3.0,
    "median": 3.0,
    "std": 1.414
  }
}

Multiple Inputs

{
  "input": {
    "x": [1.0, 2.0, 3.0],
    "y": [4.0, 5.0, 6.0],
    "alpha": 0.05
  },
  "output": 2.5
}

Floating Point Comparison

Configure tolerance in .structyl/config.json:

{
  "tests": {
    "comparison": {
      "float_tolerance": 1e-9,
      "tolerance_mode": "relative"
    }
  }
}

Tolerance Modes

Mode	Use When
relative	General purpose (default)
absolute	Comparing small values near zero
ulp	Need exact IEEE precision control

Special Values

Handle special floating point values in JSON:

{
  "input": { "x": [1.0, "Infinity", "-Infinity"] },
  "output": "NaN"
}

By default, NaN == NaN is true. Change with:

{
  "tests": {
    "comparison": {
      "nan_equals_nan": false
    }
  }
}

Binary Data

For binary data like images, use file references:

{
  "input": {
    "data": { "$file": "input.bin" }
  },
  "output": { "$file": "expected.bin" }
}

Store binary files alongside the JSON:

tests/
└── image-processing/
    ├── resize.json
    ├── input.bin
    └── expected.bin

Binary outputs are compared byte-for-byte (no tolerance).

Configuration

Full test configuration:

{
  "tests": {
    "directory": "tests",
    "pattern": "**/*.json",
    "comparison": {
      "float_tolerance": 1e-9,
      "tolerance_mode": "relative",
      "array_order": "strict",
      "nan_equals_nan": true
    }
  }
}

Field	Default	Description
directory	"tests"	Test data directory
pattern	"**/*.json"	Glob pattern for test files
float_tolerance	1e-9	Numeric comparison tolerance
tolerance_mode	"relative"	How tolerance is applied
array_order	"strict"	Whether array order matters
nan_equals_nan	true	NaN equality behavior

Running Tests

Run tests for all languages:

structyl test

Run tests for a specific language:

structyl test py
structyl test rs

Implementing Test Loaders

Each language implementation needs a test loader. Here's a simple pattern:

Python

import json
from pathlib import Path

def load_tests(suite: str) -> list[dict]:
    tests_dir = Path("tests") / suite
    return [
        json.loads(f.read_text())
        for f in tests_dir.glob("*.json")
    ]

Go

func LoadTests(suite string) []TestCase {
    pattern := filepath.Join("tests", suite, "*.json")
    files, _ := filepath.Glob(pattern)
    // Load and parse each file
}

Rust

fn load_tests(suite: &str) -> Vec<TestCase> {
    let pattern = format!("tests/{}/*.json", suite);
    // Use glob and serde_json
}

Best Practices

1. Use descriptive names: empty-array.json, negative-values.json
2. Organize by feature: One suite per function/module
3. Include edge cases: Empty inputs, boundaries, special values
4. Keep tests small: One concept per test file
5. Version control tests: Track changes in git

Next Steps

• Configuration - Full configuration reference
• Commands - Running tests and other commands