MobileAIBench

A comprehensive benchmark designed to evaluate the performance and resource consumptions of Large Language Models (LLMs) and Large Multimodal Models (LMMs) for on-device use cases. Supports both iOS and Android platforms, measuring CPU, RAM, memory, and thermal usage metrics.

from benchthing import Bench

bench = Bench("mobile-ai-bench")

bench.run(
    benchmark="mobile-ai-bench",
    task_id="1",
    models=yourMobileOptimizedModels
)

result = bench.get_result("1")