Skip to content

feat(downloadSSURGO): add LAPPLY.FUN argument for parallel unzipping#472

Merged
brownag merged 5 commits into
masterfrom
feature/issue470-2
May 7, 2026
Merged

feat(downloadSSURGO): add LAPPLY.FUN argument for parallel unzipping#472
brownag merged 5 commits into
masterfrom
feature/issue470-2

Conversation

@brownag

@brownag brownag commented May 6, 2026

Copy link
Copy Markdown
Member

Adds arguments LAPPLY.FUN and LAPPLY.FUN.ARGS to downloadSSURGO()

Closes #470

Using future.apply::future_lapply() with a 8-session future plan, the OCONUS example extracts 158 ZIP files in about 3-5 seconds on my GFE. This is a massive improvement over sequential approach where massive latency in I/O for a few different reasons is a major blocker.

library(soilDB)
library(future)

plan(multisession, workers = 8)

td <- "ssurgo"

AREASYMBOLS <- SDA_query(
  "SELECT areasymbol, saverest 
   FROM sacatalog
   WHERE SUBSTRING(areasymbol, 0, 3) IN ('PR', 'VI', 'HI', 'AK', 'PW', 'GU', 'AS', 'MP');"
)$areasymbol

downloadSSURGO(
  areasymbols = AREASYMBOLS,
  destdir = td,
  include_spatial = "mupolygon",
  include_tabular = c("mapunit", "muaggatt"),
  extract = FALSE, 
  overwrite = TRUE
) |> system.time()

downloadSSURGO(
  areasymbols = AREASYMBOLS,
  destdir = td,
  include_spatial = "mupolygon",
  include_tabular = c("mapunit", "muaggatt"),
  LAPPLY.FUN = future.apply::future_lapply,
  overwrite = TRUE
) |> system.time()

createSSURGO(
  "OCONUS.gpkg",
  exdir = td,
  overwrite = TRUE
) |> system.time()

DBI::dbListTables(con <- NASIS("OCONUS.gpkg"))

con2 <- DBI::dbConnect(duckdb::duckdb("OCONUS.duckdb"))

createSSURGO(
  conn = con2,
  exdir = td, 
  overwrite = TRUE
) |> system.time()

DBI::dbListTables(con2)
DBI::dbDisconnect(con)
DBI::dbDisconnect(con2)

@brownag brownag marked this pull request as ready for review May 7, 2026 18:27
@brownag brownag merged commit 7587460 into master May 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature requests for downloadSSURGO()

1 participant