Skip to content

README.md

Latest commit

 

History

History
382 lines (280 loc) · 11.7 KB

README.md

File metadata and controls

382 lines (280 loc) · 11.7 KB
Raw

MCTS Hyperparameter Sensitivity Studies

This directory contains systematic hyperparameter sweeps for MCTS on the U-only design space (108 compounds). Each study varies one hyperparameter while keeping others fixed at baseline values.

All studies run in thorough mode to maximize exploration and build the best possible top-N candidate list, representing a realistic DFT budget scenario.

Studies

1. Starting Material (starting_material/)

Question: How does the initial composition affect search performance?

Parameter varied: Root composition (via transition_metal and group_iv overrides)

Config Starting Material Description
cr_sn.yaml Cr₆Sn₆U Baseline (global optimum region)
fe_sn.yaml Fe₆Sn₆U Different 3d transition metal
cu_sn.yaml Cu₆Sn₆U Another 3d metal
ni_ge.yaml Ni₆Ge₆U Different metal + Group IV
w_pb.yaml W₆Pb₆U CIF default (no override)

Baseline parameters:

  • iterations: 1000
  • exploration_constant: 1.41
  • termination_limit: 100
  • rollout_depth: 3
  • move_step: 1

2. Iterations (iterations/)

Question: How does the iteration budget affect exploration and reward convergence?

Parameter varied: iterations (total MCTS iterations)

Config Iterations
iter_250.yaml 250
iter_500.yaml 500
iter_1000.yaml 1000 (baseline)
iter_2000.yaml 2000
iter_4000.yaml 4000

Fixed parameters:

  • Starting material: Cr₆Sn₆U
  • All other parameters at baseline

Note: This parameter directly controls the computational budget. For thorough mode, more iterations allow broader coverage of the design space.

3. Termination Limit (termination_limit/)

Question: How does node termination threshold affect exploration breadth?

Parameter varied: termination_limit (visits without improvement before node terminates)

Config Termination Limit
limit_25.yaml 25
limit_50.yaml 50
limit_100.yaml 100 (baseline)
limit_200.yaml 200
limit_500.yaml 500

Fixed parameters:

  • Starting material: Cr₆Sn₆U
  • All other parameters at baseline

4. Rollout Depth (rollout_depth/)

Question: How does rollout horizon affect reward estimates?

Parameter varied: rollout_depth (number of random moves per rollout)

Config Rollout Depth
depth_1.yaml 1
depth_2.yaml 2
depth_3.yaml 3 (baseline)
depth_5.yaml 5

Fixed parameters:

  • Starting material: Cr₆Sn₆U
  • All other parameters at baseline

Note: Higher depths mean more computation per rollout but potentially better long-term estimates.

5. N Rollout (n_rollout/)

Question: How does the number of rollout samples per expanded node affect value estimation quality?

Parameter varied: n_rollout (number of random lookahead walks per node, in addition to depth-0 evaluation)

Config N Rollout
rollout_0.yaml 0 (no lookahead, depth-0 only)
rollout_1.yaml 1
rollout_2.yaml 2 (baseline)
rollout_4.yaml 4
rollout_8.yaml 8

Fixed parameters:

  • Starting material: Cr₆Sn₆U
  • All other parameters at baseline

Note: Each rollout draws a random walk and evaluates multiple materials along the path (due to max-along-walk aggregation). Higher n_rollout means more computational cost per node but potentially better value estimates. n_rollout=0 means only the node's own direct evaluation is used.

6. Move Step (move_step/)

Question: How does periodic table jump size affect exploration?

Parameter varied: move_step (distance in periodic table for element substitution)

Config Move Step
step_1.yaml 1 (baseline)
step_2.yaml 2
step_3.yaml 3
step_5.yaml 5

Fixed parameters:

  • Starting material: Cr₆Sn₆U
  • All other parameters at baseline

Note: U-only space is small (108 compounds), so move_step=1 (adjacent elements) is typically sufficient.

Directory Structure

sensitivity/
├── starting_material/
│   ├── configs/          # 5 YAML configs (cr_sn, fe_sn, cu_sn, ni_ge, w_pb)
│   ├── results/          # Output directories (generated by runs)
│   └── run_all.sh        # Run all 5 configs
├── iterations/
│   ├── configs/          # 5 YAML configs (iter_250, 500, 1000, 2000, 4000)
│   ├── results/
│   └── run_all.sh
├── termination_limit/
│   ├── configs/          # 5 YAML configs (limit_25 through limit_500)
│   ├── results/
│   └── run_all.sh
├── rollout_depth/
│   ├── configs/          # 4 YAML configs (depth_1, 2, 3, 5)
│   ├── results/
│   └── run_all.sh
├── n_rollout/
│   ├── configs/          # 5 YAML configs (rollout_0, 1, 2, 4, 8)
│   ├── results/
│   └── run_all.sh
├── move_step/
│   ├── configs/          # 4 YAML configs (step_1, 2, 3, 5)
│   ├── results/
│   └── run_all.sh
├── figures/              # Generated sensitivity plots (created by plot_sensitivity.py)
├── plot_sensitivity.py   # Generate all 6 sensitivity figures
└── README.md             # This file

Running the Studies

Prerequisites

  1. Install framework with dependencies:

    cd mcts-materials
pip install -e ".[intermetallic,viz]"
# or with uv:
uv sync --extra intermetallic --extra viz

  2. Ensure data files are in repo root:

    • high_throughput_mace_results.full.csv
    • doscar_peaks_data_with_U.csv

  3. Set Materials Project API key:

    export MP_API_KEY="your-key-here"
# or use the helper script in parent directory
source ../set_api_key.sh

Running Individual Studies

# Run all configs for one study
cd sensitivity/starting_material
bash run_all.sh

# Or run individual configs
cd sensitivity/termination_limit
mcts-run run --config configs/limit_25.yaml
mcts-run run --config configs/limit_50.yaml
# ... etc

Running All Studies

cd sensitivity

# Run all 28 configs (5+5+5+4+5+4) sequentially
for study in starting_material iterations termination_limit rollout_depth n_rollout move_step; do
  cd $study
  bash run_all.sh
  cd ..
done

# Or run all studies in parallel (faster, requires more CPU cores)
for study in starting_material iterations termination_limit rollout_depth n_rollout move_step; do
  (cd $study && bash run_all.sh) &
done
wait

Note: Each config runs 1000 iterations (except the iterations study which varies this). Total runtime depends on hardware but expect several hours for all 28 runs.

Generating Figures and Analysis

After all runs complete, generate the 6 sensitivity figures and summary statistics:

cd sensitivity
python plot_sensitivity.py

This creates figures/ directory with:

  • starting_material_sensitivity.png (3"×3", 300 DPI)
  • iterations_sensitivity.png
  • termination_limit_sensitivity.png
  • rollout_depth_sensitivity.png
  • n_rollout_sensitivity.png
  • move_step_sensitivity.png
  • summary_statistics.csv (all metrics in tabular format)

Each figure plots unique compounds explored (x-axis) vs. best reward found (y-axis) for all configs in that study, with reference lines showing:

  • 90% and 95% of best reward (dashed/dotted gray lines)
  • 50% coverage marker (54 compounds, red dashed line)
  • Full design space x-axis (0-108 compounds)

Sample Efficiency Analysis

To analyze whether MCTS is effectively guiding search (vs. exhaustive exploration):

cd sensitivity
python analyze_efficiency.py

This generates detailed efficiency metrics:

  • Compounds to 90%/95%: How many evaluations to reach near-optimal reward
  • Coverage at convergence: % of design space explored when reward plateaus
  • Early performance: Best reward in first 20/50 compounds (fast/thorough mode proxies)
  • Over-exploration detection: Flags configs with >75% coverage
  • Sample efficiency ratings: EXCELLENT (<25% coverage), GOOD (<50%), MARGINAL (<75%), POOR (>75%)

Output saved to figures/efficiency_metrics.csv.

Baseline Parameters

The baseline configuration (used when a parameter is not being varied):

mcts:
  iterations: 1000
  exploration_constant: 1.41      # Standard UCB1
  termination_limit: 100
  rollout_depth: 3
  n_rollout: 2
  search_mode: thorough

intermetallic:
  transition_metal: Cr            # Start at Cr₆Sn₆U
  group_iv: Sn
  f_block_mode: u_only
  move_step: 1
  rollout_method: ehull_rdos_product
  beta: 1.0
  gamma: 1.0

Expected Outcomes

Starting Material

  • Cr₆Sn₆U and similar 3d metals (Fe, Cu, Ni) should perform well (near global optimum)
  • W₆Pb₆U may require more exploration to reach Cr-rich space

Iterations

  • 250-500: May find good materials quickly but limited exploration
  • 1000: Baseline, reasonable trade-off between compute and coverage
  • 2000-4000: Thorough exploration, better top-N list, diminishing returns after exhaustion
  • In thorough mode, higher iterations = more comprehensive coverage until space is exhausted

Termination Limit

  • Too low (25): premature termination, narrow exploration
  • Too high (500): wasted iterations revisiting exhausted nodes
  • Sweet spot likely around 100-200 for this design space

Rollout Depth

  • Depth 1: fast but myopic
  • Depth 3: proven optimal for intermetallic studies
  • Depth 5: slower, may not improve quality for small space (108 compounds)

N Rollout

  • 0: Fastest, uses only direct node evaluation (greedy)
  • 1-2: Moderate lookahead, proven effective
  • 4-8: More robust value estimates but higher cost per node
  • Trade-off: computational cost vs. value estimate quality

Move Step

  • Step 1: fine-grained, explores adjacent elements thoroughly
  • Step 3-5: larger jumps, may miss optimal neighbors in small space
  • U-only space is small enough that step 1 should suffice

Analysis Tips

Sample Efficiency (Primary Metric)

The key question: Does MCTS outperform random/exhaustive search?

  • EXCELLENT (<25% coverage to 95% reward): MCTS is highly effective, finds near-optimal in <27 compounds
  • GOOD (25-50% coverage): MCTS provides clear benefit, <54 compounds to 95% reward
  • MARGINAL (50-75% coverage): Some guidance but limited advantage over random sampling
  • POOR (>75% coverage): Essentially exhaustive search, MCTS adds little value

For the U-only space (108 compounds), good MCTS should reach 95% of best reward in <40 compounds (<37% coverage).

Additional Metrics

  1. Early performance (fast mode): Best reward in first 20 compounds
  2. Final performance (thorough mode): Best reward achieved after full budget
  3. Convergence rate: Slope of learning curves (steep = efficient exploration)
  4. Robustness: Consistent performance across different starting materials or parameter settings

Interpretation Guidelines

Fast Mode Success: Best material found in <25 compounds (<23% coverage) Thorough Mode Success: Top-10 list locked in by <50 compounds (<46% coverage)

If MCTS requires >80 compounds (>74% coverage), it's not adding value over high-throughput screening.

Notes

  • All studies use seed: 0 for reproducibility
  • search_mode: thorough ensures full iteration budget is used (no early stopping at root termination)
  • U-only design space has 108 compounds, so results may differ for larger spaces (e.g., lanthanide+U with 1620 compounds)
  • Results are saved in each study's results/<config_name>/ directory

Citation

If you use these sensitivity studies, please cite:

[Your publication info here]

© 2026. Triad National Security, LLC. All rights reserved.