This repository contains DICOM de-identification datasets, notebooks, metrics, and visual results for evaluating John Snow Labs Visual NLP across comparison workflows and synthetic dataset workflows.
| Path | Description |
|---|---|
Presidio/ |
Dataset generation and Visual NLP vs Presidio evaluation. |
Pixels_Platform/ |
Visual NLP vs Databricks Pixels evaluation on the MIDI-B dataset. |
Synthetic_V2/ |
Synthetic dataset with pixel PHI, PDF-encapsulated DICOM files, metadata PHI, and ground truth files. |
| Experiment | Dataset | Pipeline | Details |
|---|---|---|---|
| Visual NLP vs Presidio | Synthetic text overlay DICOM dataset | pre-MIDI-Pixel Visual NLP pixel de-identification pipeline | Presidio README |
| Visual NLP vs Databricks Pixels | MIDI-B validation dataset | post-MIDI-Pixel Visual NLP de-identification pipeline | Pixels Platform README |
- File count: 100 files, with 10 frames per file - 1,000 total frames.
- File size: Varies from 70 MB to 400 MB per file.
- Frame scaling: Frames are scaled down by 75% after extraction.
| Role | Instance Type | GPU | vCPUs | Memory |
|---|---|---|---|---|
| Driver | m5d.16xlarge | - | 64 | 256 GB |
| Worker | g4dn.4xlarge (T4) | 16 GB | 16 | 64 GB |
| Workers | Time Taken (s) | Cost (DBU/h) |
|---|---|---|
| 2 | 4,581.85 | 16.66 |
| 4 | 2,379.57 | 22.36 |
| 6 | 1,647.01 | 28.06 |
| Resource | Description |
|---|---|
| Visual NLP DICOM Workshop | Visual NLP DICOM notebooks and workshop examples. |
| DICOM De-identification Blogpost | Step-by-step Visual NLP DICOM de-identification walkthrough. |
| Metadata De-identification | Visual NLP guide to metadata de-identification. |
| MIDI/Pseudo-PHI DICOM Paper | Scientific Data paper describing a DICOM dataset for evaluating medical image de-identification. |
| Visual NLP Skill | John Snow Labs Visual NLP de-identification skill for your LLM. |