All-scenario Ascend AI Precision Debugging Tool
🔹 [2026.03.28]: Notice of Deprecation: ADump Module in the msProbe Repository
🔹 [2026.03.20]: Released the Foundation Model Training Accuracy Debugging Guide, Foundation Model Inference Accuracy Debugging Guide, and Common Framework Tool Instructions
🔹 [2025.12.31]: Released the open-source version of MindStudio Probe
MindStudio Probe (msProbe) is a full-scenario precision debugging toolchain for Ascend AI processors. Designed for precision debugging during model development, it supports mainstream frameworks such as PyTorch and MindSpore, helping you significantly improve the efficiency of locating model precision problems.
| Scenario | Sub-mode/Sub-scenario | Function | Description | Reference |
|---|---|---|---|---|
| vLLM Inference | eager/graph mode | Data collection | Collect msProbe precision data. | Data Collection |
| Data comparison | Compare the precision of the data dumped by msProbe to locate precision issues. | Graph Comparison in Hierarchical Visualization Precision Comparison |
||
| TorchAir graph mode | Data collection | Collect precision data by using the set_ge_dump_config API. | Data Collection | |
| Precision comparison | Compare the precision of the data dumped by msProbe to locate precision issues. | Precision Comparison | ||
| SGLang inference | eager mode | Data collection | Collect msProbe precision data. | Data Collection |
| Data comparison | Compare the precision of the data dumped by msProbe to locate precision issues. | Graph Comparison in Hierarchical Visualization Precision Comparison |
||
| ATB inference | - | Data collection | Before running an ATB model, load the ATB dump module to collect the precision data during the running of the ATB model. | Data Collection |
| Precision comparison | Compare the precision of the data dumped by ATB to locate precision issues. | Precision Comparison | ||
| Data conversion | Convert the precision data dumped by ATB into a NumPy (.npy) or PyTorch tensor (.pt) file. | Data Conversion | ||
| Offline model inference | - | Data collection | Collect msProbe precision data. | Data Collection |
| Precision comparison | Provide one-click offline model comparison by simply inputting a model without data collection in advance and generate results quickly. | Precision Comparison | ||
| Offline model data precision comparison | Compare the precision of an offline model by inputting the dump data of the offline model. | Offline Model Data Precision Comparison | ||
| Data conversion | Convert the dump data of an offline model into a NumPy (.npy) or PyTorch tensor (.pt) file. | Data Conversion | ||
| PyTorch training | - | Configuration check before training | Before training or precision comparison, compare the configuration differences that may affect training precision in the two environments. | Configuration Check Before Training |
| Data collection | Configure the config.json file to collect msProbe precision data. | Data Collection | ||
| Precision pre-check | Scan all APIs in a training model running on Ascend NPUs and provide diagnostic and analytical insights into precision. | Precision Pre-check | ||
| Graph comparison in hierarchical visualization | Parse the precision data dumped by msProbe to restore the model graph structure and compare the precision data of each model layer. | Graph Comparison in Hierarchical Visualization | ||
| Precision comparison | Compare the precision of the data dumped by msProbe to locate precision issues. | Precision Comparison | ||
| Training status monitoring | Collect and aggregate the intermediate values of the network layer, optimizer, and communication operators during model training, helping diagnose exceptions that occur during computing, communication, and optimization. | Training Status Monitoring | ||
| Checkpoint comparison | During or after training, compare two different checkpoints to evaluate model similarity. | Checkpoint Comparison | ||
| First network overflow/underflow node analysis | In the multi-rank scenario, find the first node where NaN or INF occurs through data dumping. | First Network Overflow/Underflow Node Analysis | ||
| Trend visualization | Visualize the data collected by msProbe or the training status monitoring statistics in terms of the number of iterations, rank, and tensor. | Trend Visualization | ||
| MindSpore training | - | Configuration check before training | Before training or precision comparison, compare the configuration differences that may affect training precision in the two environments. | Configuration Check Before Training |
| Data collection | Configure the config.json file to collect msProbe precision data. | Data Collection | ||
| Precision pre-check | Scan all APIs in a training model running on Ascend NPUs and provide diagnostic and analytical insights into precision. | Precision Pre-check | ||
| Graph comparison in hierarchical visualization | Parse the precision data dumped by msProbe to restore the model graph structure and compare the precision data of each model layer. | Graph Comparison in Hierarchical Visualization | ||
| Precision comparison | Compare the precision of the data dumped by msProbe to locate precision issues. | Precision Comparison | ||
| Training status monitoring | Collect and aggregate the intermediate values of the network layer, optimizer, and communication operators during model training, helping diagnose exceptions that occur during computing, communication, and optimization. | Training Status Monitoring | ||
| Overflow/Underflow detection and parsing | Overflow/Underflow detection collects precision data from APIs/modules with overflow/underflow issues, while overflow/underflow analysis examines this data to determine whether the phenomenon is normal. | Overflow/Underflow Detection and Parsing Data Collection |
||
| Checkpoint comparison | During or after training, compare two different checkpoints to evaluate model similarity. | Checkpoint Comparison | ||
| Trend visualization | Visualize the data collected by msProbe or the training status monitoring statistics in terms of the number of iterations, rank, and tensor. | Trend Visualization | ||
| MSAdapter scenario | - | Data collection | Configure the config.json file to collect msProbe precision data. | Data Collection |
| Checkpoint comparison | During or after training, compare two different checkpoints to evaluate model similarity. | Checkpoint Comparison |
An executable sample is provided to help you quickly get started with precision data collection and comparison. For details, see Quick Start of msProbe in the PyTorch Scenario or Quick Start of msProbe in the MindSpore Scenario.
msProbe supports PyPI installation, WHL installation, and source code compilation. For details, see the msProbe Installation Guide.
msProbe supports various scenarios including training and inference. Select your scenario in the Functions section above, choose the corresponding feature, and refer to the linked documentation for detailed configuration.
🔹 Foundation Model Training Accuracy Debugging Guide
🔹 Foundation Model Inference Accuracy Debugging Guide
🔹 Common Framework Tool Instructions
🔹 Precision Data Collection Baseline Report in PyTorch
🔹 Precision Pre-check Baseline Report in MindSpore
🔹 Precision Data Collection Baseline Report in MindSpore
🔹 Standard Performance Baseline Report
For frequently asked questions and solutions, see the FAQ.
To improve documentation efficiency, we provide multiple search options:
🔹 Full-text Search (ReadTheDocs): Keyword-based full-text search for interfaces, parameters, and error messages.
🔹 AI Q&A (DeepWiki): Natural language Q&A for a quick understanding of project architecture and module relationships.
🔹 AI Q&A (ZRead): Chinese Q&A with better user experience, pinpointing feature usage and details.
We welcome contributions. See the Contributing Guide.
🔹 Developer Guide
🔹 Security Statement
🔹 Disclaimer
🔹 License Declaration
You are welcome to contribute to the community. If you have any questions or suggestions, please submit Issues. We will reply as soon as possible. Thank you for your support.
| Instant Interaction (WeChat Group) | Official News (WeChat Account) | In-depth Support (Assistant/Forum) |
|---|---|---|
 Scan the QR code to join the group |
 Follow the official account |
Scan the QR codes above to join the WeChat group and follow the official account — the fastest way to connect with MindStudio users and developers: Quick Q&A: Discuss technical issues with community members Stay Updated: Receive release notes and feature update notifications Share Experience: Exchange best practices with developers More channels: 👉 Ascend Assistant:  👉 Ascend Forum:  |
msProbe is jointly developed by the following Huawei departments:
🔹 Ascend Computing MindStudio Development Department
🔹 Parallel Distributed Computing Laboratory
Thank you to everyone in the community for your PRs. We warmly welcome contributions to msProbe!