evaluating-llms-harness

This skill benchmarks code generation models using industry-standard evaluations like HumanEval and MBPP across multiple programming languages. It calculates pass@k metrics for comparing model performance, testing multi-language support, and measuring code quality. Developers should use it when rigorously evaluating or comparing coding models, as it's the same tool powering HuggingFace's code leaderboards.

LangSmith provides LLM observability for tracing, evaluating, and monitoring AI applications. Developers should use it for debugging prompts and chains, systematic output evaluation, and monitoring production systems. Its key capabilities include performance tracing, dataset testing, and analysis of latency and token usage.

Phoenix is an open-source AI observability platform for tracing, evaluating, and monitoring LLM applications. It provides detailed traces for debugging, runs evaluations on datasets, and offers real-time monitoring for production systems. Key capabilities include experiment pipelines and self-hosted observability without vendor lock-in.

This skill runs standardized code generation benchmarks like HumanEval and MBPP to evaluate model performance using pass@k metrics. It's the industry-standard tool from the BigCode Project for comparing coding abilities, testing multi-language support, and measuring code quality. Use it when benchmarking models, comparing their capabilities, or replicating HuggingFace leaderboard evaluations.