ImageNet-D, a new synthetic test set, is designed to rigorously evaluate the robustness of neural networks. It addresses limitations in existing datasets by incorporating diverse variations such as backgrounds, textures, and materials, making it more comprehensive than previous benchmarks. The dataset leverages diffusion models to generate images, which are then filtered through hard image mining to select challenging examples that expose shared failures across multiple vision models. Human-in-the-loop quality control ensures the accuracy of the dataset, with human annotators verifying the validity and correctness of the generated images. ImageNet-D offers a more robust evaluation framework, providing valuable insights into neural network understanding and helping researchers develop more robust models for real-world applications.