what would it take to support linux-binary?

[r2evans]

What is the effort required to be able to support a limited scope of linux-binary packages? By "limited scope", I mean a specific targeted linux distro and R-version combination.

As an example, I can devtools::build(..., binary=TRUE), and it will create a package (with already-compiled libraries when there is cpp code to compile) named something like mypkg_1.2.3_R_x86_64-pc-linux-gnu.tar.gz. What is not told in that filename is that it is compiled on ubuntu-24.04 using R-4.4. The embedded DESCRIPTION file reflects

Built: R 4.4.3; x86_64-pc-linux-gnu; 2025-05-23 20:29:52 UTC; unix

I can update that package into my local repo, update the PACKAGES using tools::write_PACKAGES(), and I can see in the top-level PACKAGES that the packages are found and listed with

Path: bin/x86_64-pc-linux-gnu-library/contrib/4.4

What I don't know (in the PACKAGES format) is if they are tagged as being binary somehow. (I cannot find a reference for PACKAGES's structure.) It is not served when requested, though.

Part of the complexity of serving to multiple distros is matching dependency libraries and compilers and several other things. I don't want to address any of that, since my case is to support linux-binary packages for the only linux environment that has access to my locally-served R repo.

I know that part of it is technically feasible, since p3m.dev does it (while adding the need for an HTTP user-agent string to specify the os/distro details). They have a successful business model with binary packages. I am not trying to reproduce (or compete with) the much larger scale and coverage that P3M supports.

[remlapmot]

(Am just a user)

In a very simplistic way I once had a look at this - https://remlapmot.github.io/post/2022/make-linux-binary-cran-like-repo/ - it's much simpler than you think.

As far as I know PACKAGES doesn't indicate whether they are binary packages or not - R only knows this after it has extracted the .tar.gz file. So for a repo of Linux binary packages it just has the same directory structure as for a repo of source packages (i.e., using src/contrib subdirectories). This is why for some distros P3M.dev can at times have some binary packages and some source packages within the .tar.gz files for a distro. And you can see this directory structure if you hover your mouse over any Linux binary package in an r-universe, i.e.,

With respect to the distros, P3M.dev again has a super simple solution - there is just a different URL for each distro, as shown on the setup package, e.g.,

https://packagemanager.posit.co/client/#/repos/cran/setup

for Ubuntu Noble Numbat is the CRAN URL is

https://packagemanager.posit.co/cran/__linux__/noble/latest

and for Ubuntu Jammy Jellyfish the CRAN URL is

https://packagemanager.posit.co/cran/__linux__/jammy/latest

with each URL having src/contrib beneath it (admittedly P3M is harder to inspect because it's probably a professional nginx type system).

[r2evans]

Thanks @remlapmot. I get that, and if I look internally at the PACKAGES files within both src/contrib and bin/.../contrib, I see separate PACKAGES for each of the source and binary subtrees, and the parent PACKAGES file above both src and bin has them combined. So they should be discoverable.

I think I need to confirm the problem a little bit more: the problem for me is that an on-prem GItlab instance uses docker-based CI for building and testing of R packages, and that component is not using the binary packages available in the local repo. I know the local repo is available, since non-public packages are being found and installed; I think that the order of the repos is correct, I'll verify this. Perhaps there's another component (instead of hosting/managing) that is causing a problem here.

Bottom line, I think I understood it correct, and your description confirmed what I suspected. Thanks!

[r2evans]

Actually, the issue may very well be outside of miniCRAN, but I'll post this here hoping that miniCRAN does this correctly and I've made a mistake somewhere.

From a host that has access to the private repo, I can do

> ap <- available.packages("https://my.company.com/rrepo", type="binary", ignore_repo_cache=TRUE) # ignore just in case
> ap[rownames(ap) %in% c("s2","sf"),]
    Package Version  Priority Depends                 Imports                                                                                                                         LinkingTo
 s2 "s2"    "1.1.9"  NA       "R (>= 3.0.0)"          "Rcpp, wk (>= 0.6.0)"                                                                                                           "Rcpp, wk"
 sf "sf"    "1.0-21" NA       "methods, R (>= 3.3.0)" "classInt (>= 0.4-1), DBI (>= 0.8), graphics, grDevices, grid,\nmagrittr, s2 (>= 1.1.0), stats, tools, units (>= 0.7-0), utils" "Rcpp"
    Suggests                                                                                                                                                                                                          \
                                                                                                                                                                                                                      \

 s2 "bit64, testthat (>= 3.0.0), vctrs"                                                                                                                                                                               \
                                                                                                                                                                                                                      \

 sf "blob, nanoarrow, covr, dplyr (>= 1.0.0), ggplot2, knitr,\nlwgeom (>= 0.2-14), maps, mapview, Matrix, microbenchmark,\nodbc, pbapply, pillar, pool, raster, rlang, rmarkdown,\nRPostgres (>= 1.1.0), RPostgreSQL, \
 RSQLite, sp (>= 1.2-4),\nspatstat (>= 2.0-1), spatstat.geom, spatstat.random,\nspatstat.linnet, spatstat.utils, stars (>= 0.6-0), terra,\ntestthat (>= 3.0.0), tibble (>= 1.4.1), tidyr (>= 1.2.0),\ntidyselect (>= 1\
 .0.0), tmap (>= 2.0), vctrs, wk (>= 0.9.0)"
    Enhances License                      License_is_FOSS License_restricts_use OS_type Archs MD5sum                             NeedsCompilation File
 s2 NA       "Apache License (== 2.0)"    NA              NA                    NA      NA    "4b28ecd87ea0263896d941a1967a7bb7" "yes"            NA
 sf NA       "GPL-2 | MIT + file LICENSE" NA              NA                    NA      NA    "d615956b09c840cbca0bab85f075f583" "yes"            NA
    Repository
 s2 "https://my.company.com/rrepo//bin/x86_64-pc-linux-gnu-library/contrib/4.3"
 sf "https://my.company.com/rrepo//bin/x86_64-pc-linux-gnu-library/contrib/4.3"
> install.packages("s2", repo="https://my.company.com/rrepo/", ignore_repo_cache=T)
 Installing package into ‘/home/r2/work/renv/library/R-4.3/x86_64-pc-linux-gnu’
 (as ‘lib’ is unspecified)
 Warning message:
 package ‘s2’ is not available for this version of R

 A version of this package for your version of R might be available elsewhere,
 see the ideas at
 https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages
> R.version
                _
 platform       x86_64-pc-linux-gnu
 arch           x86_64
 os             linux-gnu
 system         x86_64, linux-gnu
 status
 major          4
 minor          3.2
 year           2023
 month          10
 day            31
 svn rev        85441
 language       R
 version.string R version 4.3.2 (2023-10-31)
 nickname       Eye Holes

So the bin/x86.../contrib/4.4 tells me that the binary packages are found, whereas other source packages have "https://my.company.com/rrepo//src/contrib". However, I think it's failing because I have the binary already-compiled package as bin/.../contrib/4.3/sf_...gz but local R knows it's linux and therefore will only look in src/contrib/. The way that p3m.dev gets around this is to put binary packages into its per-distro src/contrib/, in a sense lying to the initial downloader (since the installation code recognizes the file as a binary package and installs it appropriately.

In the crux of it, I think this is a shortcoming in install.packages and R in general that it does not support the generic method of installing binary packages on linux (which is a notion supported by much complexity, I do understand the issues), though I cannot blame them for not making it easy, otherwise there would be many many people complaining because they do are pulling binary packages for the wrong distro/version of linux.

I think my path forward (whether using minCRAN or not) is to have another repo that (1) is solely setup for my preferred linux distro, easy for me; and (2) similarly "lies" by putting binary package under its src/contrib path.

Thanks for the discussion.