Which AI model gives you the best value for your task?

DSGym: A Holistic Framework for Evaluating and Training Data Science Agents

Researchers introduced DSGym, a standardized framework for evaluating and training data science agents that addresses fragmentation in existing benchmarks. The framework provides modular architecture enabling easy addition of tasks, agent scaffolds, and tools, operating as a live extensible testbed rather than static benchmarks. A key finding: many existing benchmark tasks can be solved without using actual data, which DSGym addresses through quality filtering and shortcut solvability screening. DSGym-Tasks consolidates and refines existing benchmarks. The framework expands coverage with DSBio (bioinformatics tasks grounded in research literature) and DSPredict (prediction tasks spanning computer vision, molecular prediction, and single-cell perturbation). Beyond evaluation, DSGym includes execution-verified data synthesis for agent training. In a case study, researchers trained a 4-billion-parameter model on 2,000 synthetic examples and reported it outperformed GPT-4o on standardized analysis benchmarks. The framework enables end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific contexts, addressing limitations in existing fragmented evaluation approaches.

moonshotai/Kimi-K2.5 · Hugging Face

Kimi-K2.5 is a multimodal language model released by Moonshot AI, available on Hugging Face and multiple inference providers. The model supports image-text-to-text tasks with tool-calling and structured output capabilities. It has been evaluated on multiple benchmarks: SWE-Bench Verified (70.8%), SWE-Bench Pro (50.7%), GPQA Diamond (87.6%), MMLU-Pro (87.1%), MathArena HMMT Feb 2026 (87.12%), and MDPBench with 77.5% overall performance across multilingual document understanding tasks. The model achieves 78.5% on MMMU-Pro vision benchmarks. Inference is available through multiple providers including Novita, Fireworks AI, Featherless AI, and DeepInfra, with token throughput ranging from 39–102 tokens per second depending on provider. DeepInfra offers the cheapest pricing at $2.25 per million output tokens. The model has been downloaded over 14 million times.

deepseek-ai/DeepSeek-OCR-2 · Hugging Face

DeepSeek-OCR-2, a 3 billion parameter vision-language model for optical character recognition and document understanding, is now available on Hugging Face. The model uses a causality-informed visual encoding approach and is optimized for multimodal document parsing. DeepSeek-OCR-2 scores 76.3 on the allenai/olmOCR-bench overall benchmark, with 82 on ArXiv math, 90.7 on long tiny text recognition, and 77.4 on table parsing. On the llamaindex/ParseBench dataset, it achieves 82 on text content extraction, 54 on text formatting, and 61.7 on table detection. The model weights 3.4 billion parameters in bfloat16 format and is distributed under Apache 2.0 license. Inference requires Python 3.12.9, CUDA 11.8, PyTorch 2.6.0, and Transformers 4.46.3, with optional flash-attention-2 for acceleration. The implementation supports dynamic resolution up to 1024×1024 pixels and can output structured markdown from documents. Code examples demonstrate integration via Hugging Face transformers library and vLLM for inference.

GitHub - NVlabs/vibetensor: Our first fully AI generated deep learning system · GitHub

NVIDIA Labs released VibeTensor, an open-source deep learning runtime system entirely generated by AI coding agents. The system spans Python and Node.js/TypeScript frontends down to CUDA kernel management, comprising approximately 60k non-blank lines of C++/CUDA code and 50k lines of tests. VibeTensor implements a PyTorch-inspired eager execution model with its own tensor storage, dispatcher, reverse-mode autograd, stream-ordered CUDA allocator with caching, and support for multi-GPU execution via experimental Fabric tensors. The project demonstrates that LLM-powered agents can generate coherent, testable systems code when guided by high-level architecture prompts and constrained by build/test validation rather than per-change code review. The authors validated end-to-end training on three small workloads spanning computer vision and language modeling, including multi-GPU paths. The system is released as a research artifact and explicitly not recommended for production use; performance is correctness-optimized rather than competitive with PyTorch, and the authors acknowledge composition-level inefficiencies (e.g., global serialization in hot paths). The codebase includes extensibility mechanisms: Python library overrides, C/CUDA plugin loading via stable ABI, Triton kernel bridging, and DLPack interop with other frameworks.

GitHub - facebookresearch/gcm: GPU Cluster Monitoring (GCM): Large-Scale AI Research Cluster Monitoring · GitHub

Meta Research released GPU Cluster Monitoring (GCM), an open-source monitoring toolkit for high-performance computing clusters used in AI research. The project comprises three main components: a monitoring system that collects cluster statistics from the Slurm workload scheduler to track job performance and resource utilization; health checks that verify hardware, software, network, storage, and service functionality throughout job lifecycles; and a telemetry processor that correlates OpenTelemetry data with Slurm metadata to attribute GPU utilization and other metrics to specific jobs and users. Meta uses GCM to monitor hundreds of thousands of GPUs across its AI research infrastructure. The monorepo is licensed under MIT (primary components) and Apache 2.0 (slurmprocessor), with documented roadmap items including support for additional GPU accelerators (AMD, Intel, custom), other workload schedulers beyond Slurm, and new health check types. The project includes active maintenance by Meta engineers and is open to external contributions.

GitHub - karpathy/autoresearch: AI agents running research on single-GPU nanochat training automatically · GitHub

Andrej Karpathy released autoresearch, an open-source framework that automates neural network research using AI agents. The system assigns an AI agent (Claude, Codex, or similar) to autonomously modify and iterate on a single training file over a fixed 5-minute budget per run, evaluating changes against a validation metric (bits per byte). The setup uses a simplified single-GPU implementation of nanochat training, enabling approximately 12 experiments per hour or ~100 overnight runs. The agent modifies only train.py (model architecture, hyperparameters, optimizer, batch size) while developers program high-level research direction via a markdown file (program.md). Training is constrained to exactly 5 minutes regardless of platform, ensuring fair comparison across architectural changes and automatic optimization for the target hardware. The project is self-contained, requiring only PyTorch and minimal dependencies, and currently targets single NVIDIA GPUs (tested on H100). The repository includes guidance for adapting to smaller platforms (MacBooks, Windows, AMD) via hyperparameter tuning and dataset selection, with community forks already available for macOS (MLX), Windows (RTX), and AMD support.

Building a Scaffold-Split Random Forest QSAR Co-Scientist for EGFR Inhibitor Discovery Using ChEMBL, RDKit, SHAP, and BRICS - MarkTechPost

This tutorial walks through an end-to-end computational drug discovery pipeline for EGFR inhibitor design targeting the C797S osimertinib-resistance mutation. The workflow retrieves 9000+ IC50 bioactivity records from ChEMBL, curates them into 4000 unique molecules, and trains a Random Forest QSAR model using a scaffold split to avoid memorizing close analogs. The held-out test set achieves R²=0.75, RMSE≈0.35 pIC50 units, Spearman rho≈0.76, and ROC-AUC≈0.82 for ranking potent vs. weak inhibitors. The pipeline uses RDKit for molecular standardization and Morgan fingerprint generation, SHAP for feature importance interpretation, and BRICS for fragment-based generative design. The workflow generates 4000+ novel virtual molecules, applies multi-parameter desirability scoring (potency, drug-likeness, synthetic accessibility, novelty), applies developability gates, and validates the shortlisted candidates against PubChem. The entire pipeline is demonstrated in executable Python code suitable for Google Colab, combining public APIs (ChEMBL, UniProt, PubChem) with open-source chemistry libraries.

Stop Returning Text from RAG: The Typed Answer Contract That Prevents Hallucination | Towards Data Science

This article describes a schema-based approach to reduce hallucination in retrieval-augmented generation (RAG) systems by moving from unstructured text output to typed, structured answers. The core idea is that hallucination stems from models filling gaps from training data when asked for unstructured text; constraining output to a strict schema with typed fields forces the model to ground answers in retrieved passages. The schema enforces multiple layers: typed value primitives (Amount, DateValue, TableValue, Address), multi-element answers with multi-span citations to handle non-contiguous evidence, self-assessment fields (confidence, caveats, extraction_method), and pipeline-feedback signals (answer_found, complete_answer_found, conflicting_evidence, suggested_clarification). Each field is a specific question the model answers, making every answer checkable. The article includes worked examples (extracting amounts, addresses, dates) and shows how structured extraction in RAG outperforms delegating computation to the LLM. It also covers the split between model-generated answers and pipeline-computed completeness signals, with retrieval configured to include overlap pages that let the pipeline detect when a list was truncated. Code uses Pydantic v2 and OpenAI's structured outputs API.