INTELLIGENT SCHEDULER
The intelligent scheduler
that makes every inference cheaper, faster and compliant.
Semantic cache + dynamic batching + multi-cloud routing + two-way moderation, behind an OpenAI-compatible API. Zero migration cost, 20–40% lower spend.
Overall architecture
Five layers, an end-to-end enterprise inference pipeline.
Core components
Four engines, channeling the engineering of Fireworks.ai.
Semantic cache engine
Not string matching — embedding-similarity caching. Milvus / Weaviate vector search backed by Redis. Threshold 0.95, instant hits, zero GPU cost.
- — Four-tier cache: Exact / Semantic / Template / System
- — Hit rate 30–50%, latency <10ms (vs 200–2000ms)
- — TTL + model-version invalidation
Dynamic batch optimizer
Adaptive batch sizing merges compatible requests by model, max_tokens and temperature. Four strategies (Time / Size / Adaptive / Priority) with VIP priority queue.
- — GPU utilization 30% → 70%+
- — 3–5× throughput
- — Per-token cost down 40–60%
Unified API surface
Drop-in OpenAI Chat Completions API. Enhances it with enterprise auth, smart scheduling, audit logging and billing.
- — /v1/chat/completions full parity
- — Stream, user tagging, enterprise tiering
- — Zero migration cost
Compliance wrapper
Alibaba Green Net + Tencent Tianyu + in-house keyword filters, bidirectional moderation. Algorithm-filing IDs baked in, audit logs persisted structurally.
- — Parallel moderation across providers, aggregated
- — Algorithm filing ID auto-injected into responses
- — Audit logs meet MLPS 2.0 and CAC requirements
Performance targets
Quantifiable engineering goals.
Economics
A cost structure where everyone wins.
- — GPU utilization ~30%
- — No cache — every request is paid
- — Compliance & audit on the customer
- — Semantic cache hit: $0 GPU cost
- — Miss: GPU cost $0.0014 after batching
- — TokensChain margin $0.0006 (30% GM)
Competitive advantage
Why TokensChain.
| Dimension | Direct cloud | SiliconFlow / Infini-AI | TokensChain |
|---|---|---|---|
| Cost | Baseline | Slightly lower | 20–40% lower |
| Cache | None | Basic cache | Semantic + multi-tier |
| Batching | None | Basic batching | Dynamic adaptive batching |
| Compliance | DIY | Partial | Built-in dual moderation + audit + filing |
| Enterprise | Standard SLA | Standard SLA | Custom SLA + dedicated routing |
| Migration | Adaptation needed | Adaptation needed | Zero migration (OpenAI compatible) |
Deployment architecture
Production-grade, observable, multi-region.
Multi-region deployment
South (Guangzhou) + East (Shanghai) + North (Beijing) on Kubernetes (ACK / EKS / GKE)
Data layer
Redis Cluster (3 master / 3 replica) + Milvus (2 shards / 2 replicas) + Kafka (3 brokers)
Observability
Prometheus + Grafana + AlertManager; ELK log stack; multi-dimensional alert rules
Core value
"We don't sell GPU capacity — we're the software layer that uses it efficiently.
Plug into our API and cut costs 20–40% instantly, with compliance done for you and zero migration."