NVIDIA NeMo Curator

NeMo Curator is a data curation library for text, image, video and audio. It supports curating and preparing high-quality text datasets for LLM training (filtering, formatting, deduplication), curating image-text datasets with embedding, classification and deduplication, processing videos with GPU-accelerated pipelines and sharding, and transcribing, filtering and curating speech and audio datasets with ASR models.