packages | B101nfo

useR madrid: rtweet

Thu, 29 Feb 2024 19:00:00 +0200

This presentation was in Spanish. I shared the history of my involvement with rtweet and what is happening with the package and Twitter API.

CRAN maintained packages

Wed, 03 May 2023 00:00:00 +0000

The role of package managers in software is paramount for developers. In R the CRAN team provides a platform to tests and host packages. This means ensuring that R dependencies are up to date and software required by some packages are also available in CRAN.

This helps testing ~20000 packages frequently (daily for most packages) in several architectures and R versions. In addition, they test updates for compatibility with the dependencies and test and review new packages.

Most of the work with packages is automated but often requires human intervention (50% of the submisions). Another consuming activity is keeping up packages abandoned by their original maintainers.

While newer packages are archived from CRAN often, some old packages were adopted by CRAN. The CRAN team is looking for help maintining those.

In this post I’ll explore the packages maintained by CRAN.

CRAN in packages

packages_db <- as.data.frame(tools::CRAN_package_db())
cran_author <- grep("CRAN Team", x = packages_db$Author, ignore.case = TRUE)
cran_authorsR <- grep("CRAN Team", x = packages_db$`Authors@R`, ignore.case = TRUE)
CRAN_TEAM_mentioned <- union(cran_author, cran_authorsR)
unique(packages_db$Package[CRAN_TEAM_mentioned])
## [1] "fBasics"   "fMultivar" "geiger"    "plotrix"   "RCurl"     "RJSONIO"  
## [7] "udunits2"  "XML"

In some of these package the CRAN team appears as contributors because they provided help/code to fix bugs:

In others they are the maintainers:

From these three packages RJSONIO is the newest (first release in 2010) and requires less updates (lately 1 or 2 a year). However, in 2022 RCurl and XML required 4 and 5 updates respectively. I will focus on these packages as these are the ones they are looking for new maintainers.

RCurl and XML

Circular dependency

Both XML and RCurl depend on each other.

We can see that the packages are direct dependencies of one of their direct dependencies! How can be that? If we go the the RCurl website we see in “Suggests: XML”, and in the XML website the RCurl is there too. This circular dependency is allowed because they have each other in Suggests.

A first step to reduce any possible problem would be to separate them. This would make it easier understanding which package is worth prioritizing and possible missteps will have less impact.

If we look at XML source code for RCurl we find some code in inst/ folder. If these two cases were removed the package could remove its dependency to RCurl.

Similarly, if we look at RCurl source code for XML we find some code in inst/ folder and in some examples. If these three cases were removed the package could remove its dependency to XML.

RCurl has been more stable than XML, which have seen new functions added and one removed since CRAN is maintaining it.

Relevant data

We will look at 4 sets of data for each pacakge: dependencies, releases, maintainers and downloads.

Dependencies

Both packages have some system dependencies which might make the maintenance harder. In addition they have a large number of dependencies. We can gather the dependencies in CRAN and Bioconductor software packages:

library("tools")
# Look up only software dependencies in Bioconductor
options(repos = BiocManager::repositories()[c("BioCsoft", "CRAN")])
ap <- available.packages()
all_deps <- package_dependencies(c("RCurl", "XML"), 
                                 reverse = TRUE, db = ap, which = "all")
all_unique_deps <- unique(unlist(all_deps, FALSE, FALSE))
first_deps <- package_dependencies(all_unique_deps, db = ap, which = "all")
first_deps_strong <- package_dependencies(all_unique_deps, db = ap, which = "strong")
strong <- sapply(first_deps_strong, function(x){any(c("XML", "RCurl") %in% x)})
deps_strong <- package_dependencies(all_unique_deps, recursive = TRUE, 
                                 db = ap, which = "strong")
first_rdeps <- package_dependencies(all_unique_deps, 
                                   reverse = TRUE, db = ap, which = "all")
deps_all <- package_dependencies(all_unique_deps, recursive = TRUE, 
                                 db = ap, which = "all")

They have 495 direct dependencies (and 8 more in annotation packages in Bioconductor: recount3, ENCODExplorerData, UCSCRepeatMasker, gDNAinRNAseqData, qdap, qdapTools, metaboliteIDmapping and curatedBreastData).

These two packages with their dependencies are used one way or another by around 20000 packages (about 90% of CRAN and Bioconductor)! If these packages fail the impact on the community will be huge.

To reduce the impact of the dependencies we should look up the direct dependencies. But we also looked at the reverse dependencies to asses the impact of the package in the other packages.

Know which are these, and who maintain them will help decide what is the best course of action.

Releases

A first approach is looking into the number of releases and dates to asses if the package has an active maintainer or not:

archive <- tools:::CRAN_archive_db()[all_unique_deps]
packages <- tools::CRAN_package_db()
library("dplyr")
library("BiocPkgTools")
fr <- vapply(archive, function(x) {
  if (is.null(x)) {
    return(NA)
  }
  as.Date(x$mtime[1])
}, FUN.VALUE = Sys.Date())
fr_bioc <- biocDownloadStats("software") |> 
  filter(Package %in% all_unique_deps) |> 
  firstInBioc() |> 
  pull(Date, name = Package)
first_release <- c(as.Date(fr[!is.na(fr)]), as.Date(fr_bioc))[all_unique_deps]
last_update <- packages$Published[match(all_unique_deps, packages$Package)]
releases <- vapply(archive, NROW, numeric(1L)) + 1

We only have information about CRAN packages:
Bioconductor has two releases every year, and while the maintainers can release patched versions of packages between them that information is not stored (or easily retrieved, they are still available in the git server).

Even if Bioconductor maintainers didn’t modify the package the version number increases with each release. But the version update in the git doesn’t propagate to users automatically unless their checks pass. For all these reasons it doesn’t make sense to count releases of packages in Bioconductor.

Maintainers

Now that we know which packages are more active, we can look up for the people behind it. This way we can prioritize working with maintainers that are known to be active¹.

maintainers <- packages_db$Maintainer[match(all_unique_deps, packages_db$Package)]
maintainers <- trimws(gsub("<.+>", "", maintainers))

Once again, the Bioconductor repository doesn’t provide a file to gather this kind of data.

Downloads

Another variable we can use are the downloads from users of said packages. Probably, packages more downloaded are used more and a breaking change on them will have impact on more people than other packages.

library("cranlogs")
acd <- cran_downloads(intersect(all_unique_deps, packages_db$Package), 
                          when = "last-month")
cran_pkg <- summarise(acd, downloads = sum(count), .by = package)
loc <- Sys.setlocale(locale = "C")
bioc_d <- vapply(setdiff(all_unique_deps, packages_db$Package), function(x){
  pkg <- pkgDownloadStats(x)
  tail(pkg$Nb_of_downloads, 1)
  }, numeric(1L))
bioc_pkg <- data.frame(package = names(bioc_d), downloads = bioc_d)
downloads <- rbind(bioc_pkg, cran_pkg)
rownames(downloads) <- downloads$package
dwn <- downloads[all_unique_deps, ]

The logs are provided by the global mirror of CRAN (sponsored by Rstudio).
The Bioconductor infrastructure which provides total number of downloads and number of downloads from distinct IPs ².

Analysis

We collected the data that might be relevant. Now, we can start looking all the data gathered:

repo <- vector("character", length(all_unique_deps))
ap_deps <- ap[all_unique_deps, ]
repo[startsWith(ap_deps[, "Repository"], "https://bioc")] <- "Bioconductor"
repo[!startsWith(ap_deps[, "Repository"], "https://bioc")] <- "CRAN"
deps <- data.frame(package = all_unique_deps,
                   direct_dep_XML = all_unique_deps %in% all_deps$XML,
                   direct_dep_RCurl = all_unique_deps %in% all_deps$RCurl,
                   first_deps_n = lengths(first_deps),
                   deps_all_n = lengths(deps_all),
                   first_rdeps_n = lengths(first_rdeps),
                   first_deps_strong_n = lengths(first_deps_strong), 
                   deps_strong_n = lengths(deps_strong),
                   direct_strong = strong, 
                   releases = releases,
                   strong = strong, 
                   first_release = first_release,
                   last_release = last_update,
                   maintainer = maintainers,
                   downloads = dwn$downloads,
                   repository = repo) |> 
  mutate(type = case_when(direct_dep_XML & direct_dep_RCurl ~ "both",
                          direct_dep_XML ~ "XML",
                          direct_dep_RCurl ~ "RCurl"))
rownames(deps) <- NULL
head(deps)

package	direct_dep_XML	direct_dep_RCurl	first_deps_n	deps_all_n	first_rdeps_n	first_deps_strong_n	deps_strong_n	direct_strong	releases	strong	first_release	last_release	maintainer	downloads	repository	type
AnnotationForge	TRUE	TRUE	26	2456	5	10	47	TRUE	1	TRUE	2012-02-01	NA	NA	8113	Bioconductor	both
AnnotationHubData	TRUE	TRUE	33	2456	4	26	136	TRUE	1	TRUE	2015-02-01	NA	NA	6619	Bioconductor	both
autonomics	FALSE	TRUE	61	2499	0	34	104	FALSE	1	FALSE	2021-02-01	NA	NA	91	Bioconductor	RCurl
BaseSpaceR	FALSE	TRUE	6	2456	0	3	4	TRUE	1	TRUE	2013-02-01	NA	NA	218	Bioconductor	RCurl
BayesSpace	FALSE	TRUE	34	2459	0	24	161	TRUE	1	TRUE	2020-02-01	NA	NA	221	Bioconductor	RCurl
BgeeDB	FALSE	TRUE	19	2457	2	14	71	TRUE	1	TRUE	2016-02-01	NA	NA	238	Bioconductor	RCurl

I added some numbers and logical values that might help exploring this data.

We will look at the packages dependencies between RCurl and XML, some plots to have a quick view

Distribution dependencies

Let’s see how many packages depend in each of them:

deps |> 
  summarise(Packages = n(), deps = sum(first_deps_n),
            q25 = quantile(deps_all_n, probs = 0.25),
            mean_all = mean(deps_all_n),
            q75 = quantile(deps_all_n, probs = 0.75),
            .by = c(direct_dep_XML, direct_dep_RCurl)) |> 
  arrange(-Packages)

direct_dep_XML	direct_dep_RCurl	Packages	deps	q25	mean_all	q75
TRUE	FALSE	235	3584	2456	2365.596	2458.5
FALSE	TRUE	193	3187	2456	2320.855	2460.0
TRUE	TRUE	67	1216	2456	2423.119	2457.5

There are ~40 more packages depending on XML than to RCurl and just 67 to both of them.

Overview

We can plot some variables to get a quick overview of the packages:

library("ggplot2")
library("ggrepel")
deps_wo <- filter(deps, !package %in% c("XML", "RCurl"))
deps_wo |> 
  ggplot() +
  geom_point(aes(first_deps_n, downloads, shape = type)) +
  geom_text_repel(aes(first_deps_n, downloads, label = package),
                  data = filter(deps_wo, first_deps_n > 40 | downloads > 10^5)) +
  theme_minimal() +
  scale_y_log10(labels = scales::label_log()) +
  labs(title = "Packages and downloads", 
       x = "Direct dependencies", y = "Downloads", size = "Packages")
## Warning: ggrepel: 1 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Figure 1: Direct dependencies vs downloads. Many pakcages have up to 50 packages and most have below 1000 downloads in a month.

There is an outlier on 1, the mlr package has more than 10k downloads and close to 120 direct dependencies, but down to less than 15 strong dependencies !

deps_wo |> 
  ggplot() +
  geom_point(aes(first_deps_n, first_rdeps_n, shape = type)) +
  geom_text_repel(aes(first_deps_n, first_rdeps_n, label = package),
                  data = filter(deps_wo, first_deps_n > 60 | first_rdeps_n > 50)) +
  theme_minimal() +
  scale_y_log10(labels = scales::label_log()) +
  labs(title = "Few dependencies but lots of dependents",
    x = "Direct dependencies", y = "Depend on them", size = "Packages")
## Warning: Transformation introduced infinite values in continuous y-axis
## Transformation introduced infinite values in continuous y-axis

Figure 2: Dependencies vs packages that depend on them.

In general though, the packages that have more dependencies have less direct dependencies.

library("ggplot2")
library("ggrepel")
deps_wo <- filter(deps, !package %in% c("XML", "RCurl"))
deps_wo |> 
  ggplot() +
  geom_vline(xintercept = 20, linetype = 2) +
  geom_point(aes(first_deps_strong_n, downloads, shape = repository)) +
  geom_text_repel(aes(first_deps_strong_n, downloads, label = package),
                  data = filter(deps_wo, first_deps_strong_n > 20 | downloads > 10^5)) +
  theme_minimal() +
  scale_y_log10(labels = scales::label_log()) +
  labs(title = "Packages and downloads", 
       x = "Direct strong dependencies", y = "Downloads", shape = "Repository")
## Warning: ggrepel: 20 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Figure 3: Direct strong dependencies vs downloads. Many pakcages have more than 20 direct imports.

One observable effect is that many packages do not comply with current CRAN rules of having 20 strong dependencies (as described in R-internals). This suggests that these CRAN packages are old or that this limit is not checked in packages updates.

data_maintainers <- deps_wo |> 
  filter(!is.na(maintainer)) |> 
  summarize(n = n(), downloads = sum(downloads), .by = maintainer)
data_maintainers |> 
  ggplot() +
  geom_point(aes(n, downloads)) +
  geom_text_repel(aes(n, downloads, label = maintainer),
                  data = filter(data_maintainers, n > 2 | downloads > 10^4)) +
  scale_y_log10(labels = scales::label_log()) +
  scale_x_continuous(breaks = 1:10, minor_breaks = NULL) +
  theme_minimal() +
  labs(title = "CRAN maintainers that depend on XML and RCurl",
       x = "Packages", y = "Downloads")
## Warning: ggrepel: 15 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Figure 4: Looking at maintainers and the number of downloads they have.

Most maintainer have few packages, some highly used packages but some have many packages relatively highly used.

Finding important packages

We can use a PCA to find which packages are more important.

cols_pca <-  c(4:7, 15)
pca_all <- prcomp(deps_wo[, cols_pca], scale. = TRUE, center = TRUE)
summary(pca_all)
## Importance of components:
##                          PC1    PC2    PC3     PC4     PC5
## Standard deviation     1.386 1.2478 0.9458 0.65380 0.44846
## Proportion of Variance 0.384 0.3114 0.1789 0.08549 0.04022
## Cumulative Proportion  0.384 0.6954 0.8743 0.95978 1.00000
pca_data <- cbind(pca_all$x, deps_wo)
ggplot(pca_data) +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  geom_point(aes(PC1, PC2, col = repository, shape = repository)) +
  geom_text_repel(aes(PC1, PC2, label = package), 
                  data = filter(pca_data, abs(PC1) > 2 | abs(PC2) > 2)) +
  theme_minimal() +
  theme(axis.text = element_blank()) +
  labs(title = "PCA of the numeric variables")
## Warning: ggrepel: 58 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Figure 5: PCA of all packages.

We can see in the first PCA some packages that have many downloads and/or depend on many packages. The second one are packages with many dependencies, as explained by rotation:

pca_all$rotation[, 1:2]

	PC1	PC2
first_deps_n	-0.6521642	-0.1528947
deps_all_n	-0.3304698	-0.0549046
first_rdeps_n	0.1235972	-0.6948659
first_deps_strong_n	-0.6606765	-0.0750116
downloads	0.1170554	-0.6965223

But more important is that are packages that are named in 5, there is the RUnit package, markdown and rgeos that have high number of downloads and many package depend on them one way or another.

However we can focus on packages that without RCurl or XML wouldn’t work:

pca_strong <- prcomp(deps_wo[deps_wo$strong, cols_pca], 
                     scale. = TRUE, center = TRUE)
summary(pca_strong)
## Importance of components:
##                           PC1    PC2    PC3     PC4     PC5
## Standard deviation     1.4198 1.3005 0.9373 0.49421 0.41258
## Proportion of Variance 0.4032 0.3382 0.1757 0.04885 0.03404
## Cumulative Proportion  0.4032 0.7414 0.9171 0.96596 1.00000
pca_data_strong <- cbind(pca_strong$x, deps_wo[deps_wo$strong, ])
ggplot(pca_data_strong) +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  geom_point(aes(PC1, PC2, col = repository, shape = repository)) +
    geom_text_repel(aes(PC1, PC2, label = package), 
                  data = filter(pca_data_strong, abs(PC1) > 2 | abs(PC2) > 2)) +
  theme_minimal() +
  theme(axis.text = element_blank()) +
  labs(title = "Important packages depending on XML and RCurl", 
       subtitle = "PCA of numeric variables of strong dependencies",
       col = "Repository", shape = "Repository")
## Warning: ggrepel: 42 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Figure 6: PCA of packages with strong dependency to XML or RCurl.

The main packages that depend on XML and RCurl are from Biocondcutor, followed by mlr and rlist. rlist has as dependency XML and only uses 3 functions from it. mlr uses 5 different functions from XML.

pca_weak <- prcomp(deps_wo[!deps_wo$strong, cols_pca], 
                   scale. = TRUE, center = TRUE)
summary(pca_weak)
## Importance of components:
##                           PC1    PC2    PC3     PC4     PC5
## Standard deviation     1.4500 1.1578 0.9901 0.63980 0.40895
## Proportion of Variance 0.4205 0.2681 0.1960 0.08187 0.03345
## Cumulative Proportion  0.4205 0.6886 0.8847 0.96655 1.00000
pca_data_weak <- cbind(pca_weak$x, deps_wo[!deps_wo$strong, ])
ggplot(pca_data_weak) +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  geom_point(aes(PC1, PC2, col = type, shape = type)) +
  geom_text_repel(aes(PC1, PC2, label = package), 
                  data = filter(pca_data_weak, abs(PC1)> 2 | abs(PC2) > 2)) +
  theme_minimal() +
  theme(axis.text = element_blank()) +
  labs(title = "PCA of packages in CRAN", col = "Type", shape = "Type")

Figure 7: Packages with weak dependency to XML or RCurl.

keep <- deps_wo$repository == "CRAN" & deps_wo$strong
pca_cran <- prcomp(deps_wo[keep, cols_pca], 
                     scale. = TRUE, center = TRUE)
summary(pca_cran)
## Importance of components:
##                           PC1    PC2    PC3     PC4     PC5
## Standard deviation     1.4174 1.3060 0.9244 0.51813 0.40278
## Proportion of Variance 0.4018 0.3412 0.1709 0.05369 0.03245
## Cumulative Proportion  0.4018 0.7430 0.9139 0.96755 1.00000
pca_data_strong <- cbind(pca_cran$x, deps_wo[keep, ])
ggplot(pca_data_strong) +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  geom_point(aes(PC1, PC2, col = type, shape = type)) +
    geom_text_repel(aes(PC1, PC2, label = package), 
                  data = filter(pca_data_strong, abs(PC1) > 2 | abs(PC2) > 2)) +
  theme_minimal() +
  theme(axis.text = element_blank()) +
  labs(title = "Packages in CRAN", 
       col = "Type", shape = "Type")
## Warning: ggrepel: 26 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Figure 8: PCA of packages on CRAN.

keep <- deps_wo$repository == "Bioconductor"  & deps_wo$strong
pca_bioc <- prcomp(deps_wo[keep, cols_pca], 
                     scale. = TRUE, center = TRUE)
summary(pca_bioc)
## Importance of components:
##                           PC1    PC2    PC3     PC4     PC5
## Standard deviation     1.4913 1.3703 0.8495 0.33584 0.25281
## Proportion of Variance 0.4448 0.3755 0.1443 0.02256 0.01278
## Cumulative Proportion  0.4448 0.8203 0.9647 0.98722 1.00000
pca_data_strong <- cbind(pca_bioc$x, deps_wo[keep, ])
ggplot(pca_data_strong) +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  geom_point(aes(PC1, PC2, col = type, shape = type)) +
    geom_text_repel(aes(PC1, PC2, label = package), 
                  data = filter(pca_data_strong, abs(PC1) > 2 | abs(PC2) > 2)) +
  theme_minimal() +
  theme(axis.text = element_blank()) +
  labs(title = "Packages in Bioconductor", 
       subtitle = "PCA of numeric variables of strong dependencies",
       col = "Type", shape = "Type")
## Warning: ggrepel: 4 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Figure 9: PCA of packages on Bioconductor.

GenomeInfoDb is the package that seems more important that only uses the RCurl::getURL function.

Outro

I wanted to explore a bit how these packages got into this position ³.

deps |> 
  filter(strong) |> 
  ggplot() +
  geom_vline(xintercept = as.Date("2013-06-15"), linetype = 2) +
  geom_point(aes(first_release, downloads, col = type, shape = type, 
                 size = first_deps_strong_n)) +
  geom_label(aes(first_release, downloads, label = package),
             data = filter(deps, package %in% c("XML", "RCurl")), show.legend = FALSE) +
  theme_minimal() +
  scale_y_log10(labels = scales::label_log()) +
  annotate("text", x = as.Date("2014-6-15"), y = 5*10^5, 
           label = "CRAN maintained", hjust = 0) +
  labs(x = "Release date", y = "Downloads", 
       title = "More packages added after CRAN maintenance than before",
       subtitle = "Release date and downloads",
       col = "Depends on", shape = "Depends on", size = "Direct strong dependencies") 
## Warning: Removed 34 rows containing missing values (`geom_point()`).

Figure 10: First release of packages in relation to the maintenance by CRAN of XML and RCurl.

Almost the CRAN team have been maintaining these packages longer than the previous maintainer(s?).

Next, we look at the dependencies added after CRAN started maintaining them

summarize(deps_wo,
          before = sum(first_release <= as.Date("2013-06-15"), na.rm = TRUE), 
          later = sum(first_release > as.Date("2013-06-15"), na.rm = TRUE),
          .by = type)

type	before	later
both	14	52
RCurl	21	150
XML	63	156

More packages have been released after CRAN is maintaining it than before. Maybe packages authors trusted the CRAN team for their dependencies or there was no other alternative for the functionality. This might also be explained by the expansion of CRAN (and Bioconductor) with more packages being added each day. However, this places further pressure in the CRAN team to maintain those packages. Removing this burden might free more time for them or to dedicate to CRAN.

A replacement for XML could be xml2, first released in 2015 (which uses the same system dependency libxml2).
A replacement for RCurl could be curl, first released at the end of 2014 (which uses the same system dependency libcurl).

Until their release there were no other replacement for these packages (if there are other packages, please let me know). It is not clear to me if those packages at their first release could replace XML and RCurl.

This highlight the importance of correct replacement of packages in the community. Recent examples are the efforts taken by the spatial community led by Roger Bivand, Edzer Pebesma. Where packages have been carefully designed and planned to replace older packages that are going to be retired soon.

Recomendations

As a final recommendations I think:

Disentangle the XML and RCurl circular dependency.
Evaluate if the xml2 and curl packages provides enough functionality to replace XML and RCurl respectively. If not see what should be added to these packages or how to develop alternative packages to fill the gap if needed.
Maybe a helping documentation about the alternative from XML and RCurl could be written to ease the transition and evaluate if the functionality is covered by these packages.
Contact package maintainers to replace the functionality they currently depend on XML and RCurl as seen in 4 and the maintainers of packages seen in figures 5, 6, 8, and 9.
Set deprecation warnings on the XML and RCurl packages.
Archive XML and RCurl packages in CRAN.

This might take years of moving packages around but I am confident that once the word is out, package developers will avoid XML and RCurl and current maintainers that depend on them will replace them.

Update:

On 2024/01/22 the CRAN team asked for a maintainer of XML

Reproducibility

## - Session info ---------------------------------------------------------------
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       Ubuntu 22.04.3 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  C
##  ctype    C
##  tz       Europe/Madrid
##  date     2024-01-22
##  pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## - Packages -------------------------------------------------------------------
##  package       * version     date (UTC) lib source
##  Biobase         2.62.0      2023-10-24 [1] Bioconductor
##  BiocFileCache   2.10.1      2023-10-26 [1] Bioconductor
##  BiocGenerics    0.48.1      2023-11-01 [1] Bioconductor
##  BiocManager     1.30.22     2023-08-08 [1] CRAN (R 4.3.1)
##  BiocPkgTools  * 1.20.0      2023-10-24 [1] Bioconductor
##  biocViews       1.70.0      2023-10-24 [1] Bioconductor
##  bit             4.0.5       2022-11-15 [1] CRAN (R 4.3.1)
##  bit64           4.0.5       2020-08-30 [1] CRAN (R 4.3.1)
##  bitops          1.0-7       2021-04-24 [1] CRAN (R 4.3.1)
##  blob            1.2.4       2023-03-17 [1] CRAN (R 4.3.1)
##  blogdown        1.18        2023-06-19 [1] CRAN (R 4.3.1)
##  bookdown        0.37        2023-12-01 [1] CRAN (R 4.3.1)
##  bslib           0.6.1       2023-11-28 [1] CRAN (R 4.3.1)
##  cachem          1.0.8       2023-05-01 [1] CRAN (R 4.3.1)
##  cli             3.6.2       2023-12-11 [1] CRAN (R 4.3.1)
##  codetools       0.2-19      2023-02-01 [2] CRAN (R 4.3.1)
##  colorspace      2.1-0       2023-01-23 [1] CRAN (R 4.3.1)
##  cranlogs      * 2.1.1       2019-04-29 [1] CRAN (R 4.3.1)
##  crul            1.4.0       2023-05-17 [1] CRAN (R 4.3.1)
##  curl            5.2.0       2023-12-08 [1] CRAN (R 4.3.1)
##  DBI             1.2.1       2024-01-12 [1] CRAN (R 4.3.1)
##  dbplyr          2.4.0       2023-10-26 [1] CRAN (R 4.3.2)
##  digest          0.6.34      2024-01-11 [1] CRAN (R 4.3.1)
##  dplyr         * 1.1.4       2023-11-17 [1] CRAN (R 4.3.1)
##  DT              0.31        2023-12-09 [1] CRAN (R 4.3.1)
##  evaluate        0.23        2023-11-01 [1] CRAN (R 4.3.2)
##  fansi           1.0.6       2023-12-08 [1] CRAN (R 4.3.1)
##  farver          2.1.1       2022-07-06 [1] CRAN (R 4.3.1)
##  fastmap         1.1.1       2023-02-24 [1] CRAN (R 4.3.1)
##  fauxpas         0.5.2       2023-05-03 [1] CRAN (R 4.3.1)
##  filelock        1.0.3       2023-12-11 [1] CRAN (R 4.3.1)
##  generics        0.1.3       2022-07-05 [1] CRAN (R 4.3.1)
##  ggplot2       * 3.4.4       2023-10-12 [1] CRAN (R 4.3.1)
##  ggrepel       * 0.9.5       2024-01-10 [1] CRAN (R 4.3.1)
##  gh              1.4.0       2023-02-22 [1] CRAN (R 4.3.1)
##  glue            1.7.0       2024-01-09 [1] CRAN (R 4.3.1)
##  graph           1.80.0      2023-10-24 [1] Bioconductor
##  gtable          0.3.4       2023-08-21 [1] CRAN (R 4.3.1)
##  highr           0.10        2022-12-22 [1] CRAN (R 4.3.1)
##  hms             1.1.3       2023-03-21 [1] CRAN (R 4.3.1)
##  htmltools       0.5.7       2023-11-03 [1] CRAN (R 4.3.2)
##  htmlwidgets   * 1.6.4       2023-12-06 [1] CRAN (R 4.3.1)
##  httpcode        0.3.0       2020-04-10 [1] CRAN (R 4.3.1)
##  httr            1.4.7       2023-08-15 [1] CRAN (R 4.3.1)
##  igraph          1.6.0       2023-12-11 [1] CRAN (R 4.3.1)
##  jquerylib       0.1.4       2021-04-26 [1] CRAN (R 4.3.1)
##  jsonlite        1.8.8       2023-12-04 [1] CRAN (R 4.3.1)
##  knitr         * 1.45        2023-10-30 [1] CRAN (R 4.3.2)
##  labeling        0.4.3       2023-08-29 [1] CRAN (R 4.3.2)
##  lifecycle       1.0.4       2023-11-07 [1] CRAN (R 4.3.2)
##  magrittr        2.0.3       2022-03-30 [1] CRAN (R 4.3.1)
##  memoise         2.0.1       2021-11-26 [1] CRAN (R 4.3.1)
##  munsell         0.5.0       2018-06-12 [1] CRAN (R 4.3.1)
##  pillar          1.9.0       2023-03-22 [1] CRAN (R 4.3.1)
##  pkgconfig       2.0.3       2019-09-22 [1] CRAN (R 4.3.1)
##  purrr           1.0.2       2023-08-10 [1] CRAN (R 4.3.1)
##  R6              2.5.1       2021-08-19 [1] CRAN (R 4.3.1)
##  RBGL            1.78.0      2023-10-24 [1] Bioconductor
##  Rcpp            1.0.12      2024-01-09 [1] CRAN (R 4.3.1)
##  RCurl           1.98-1.14   2024-01-09 [1] CRAN (R 4.3.1)
##  readr           2.1.5       2024-01-10 [1] CRAN (R 4.3.1)
##  rlang           1.1.3       2024-01-10 [1] CRAN (R 4.3.1)
##  rmarkdown       2.25        2023-09-18 [1] CRAN (R 4.3.1)
##  rorcid          0.7.0       2021-01-20 [1] CRAN (R 4.3.1)
##  RSQLite         2.3.5       2024-01-21 [1] CRAN (R 4.3.1)
##  rstudioapi      0.15.0      2023-07-07 [1] CRAN (R 4.3.1)
##  RUnit           0.4.32      2018-05-18 [1] CRAN (R 4.3.1)
##  rvest           1.0.3       2022-08-19 [1] CRAN (R 4.3.1)
##  sass            0.4.8       2023-12-06 [1] CRAN (R 4.3.1)
##  scales          1.3.0       2023-11-28 [1] CRAN (R 4.3.1)
##  sessioninfo     1.2.2       2021-12-06 [1] CRAN (R 4.3.1)
##  stringi         1.8.3       2023-12-11 [1] CRAN (R 4.3.1)
##  stringr         1.5.1       2023-11-14 [1] CRAN (R 4.3.1)
##  tibble          3.2.1       2023-03-20 [1] CRAN (R 4.3.1)
##  tidyselect      1.2.0       2022-10-10 [1] CRAN (R 4.3.1)
##  tzdb            0.4.0       2023-05-12 [1] CRAN (R 4.3.1)
##  utf8            1.2.4       2023-10-22 [1] CRAN (R 4.3.2)
##  vctrs           0.6.5       2023-12-01 [1] CRAN (R 4.3.1)
##  whisker         0.4.1       2022-12-05 [1] CRAN (R 4.3.1)
##  withr           3.0.0       2024-01-16 [1] CRAN (R 4.3.1)
##  xfun            0.41        2023-11-01 [1] CRAN (R 4.3.2)
##  XML             3.99-0.16.1 2024-01-22 [1] CRAN (R 4.3.1)
##  xml2            1.3.6       2023-12-04 [1] CRAN (R 4.3.1)
##  yaml            2.3.8       2023-12-11 [1] CRAN (R 4.3.1)
## 
##  [1] /home/lluis/bin/R/4.3.1
##  [2] /opt/R/4.3.1/lib/R/library
## 
## ------------------------------------------------------------------------------

the maintainer function only works for installed packages, and I don’t have all these packages installed.↩︎
Both logs only count those of their repository and not from other mirrors or approaches (RSPM, bspm, r2u, ….).↩︎
I recently found this as opposite of introduction/intro.↩︎

experDesign: follow up

Sun, 09 Apr 2023 00:00:00 +0000

I am happy to announce a new release of experDesign. Install it from CRAN with:

install.packages("experDesign")
library("experDesign")

This new release has focused in more tricky aspects when designing an experiment:

Checking the samples of your experiment.
How to continue stratifying your conditions after some initial batch.

These functions should be used before carrying out anything once you have your samples collected. You can use these functions and make an informed decision of what might happen with your experiment.

Checking your samples

The new function check_data() will warn you if it finds some known issues with your data.

library("experDesign")
library("MASS")

If we take the survey dataset from the MASS package we can see that it has some issues:

data(survey, package = "MASS")
check_data(survey)
## Warning: Two categorical variables don't have all combinations.
## Warning: Some values are missing.
## Warning: There is a combination of categories with no replicates; i.e. just one
## sample.
## [1] FALSE

While if we fabricate our own dataset we might realize we have a problem

rdata <- expand.grid(sex = c("M", "F"), class = c("lower", "median", "high"))
stopifnot("Same samples/rows as combinations of classes" = nrow(rdata) == 2*3)
check_data(rdata)
## Warning: There is a combination of categories with no replicates; i.e. just one
## sample.
## [1] FALSE
# We create some new samples with the same conditions
rdata2 <- rbind(rdata, rdata)
check_data(rdata2)
## [1] TRUE

One might decide to go ahead with what is available or use only some of those samples or wait to collect more samples for the experiment

Follow up

Imagine you have 100 samples that you distribute in 4 batches of 25 samples each. Later, you collect 80 more samples to analyze. You want these new samples to be analyzed together with those previous 100 samples. Will it be possible? How should you distribute your new samples in groups of 25?

Using the same dataset from MASS imagine if we first collected 118 observations and later 119 more:

survey1 <- survey[1:118, ]
survey2 <- survey[119:nrow(survey), ]
# Using low number of iterations to speed the process 
# you should even use higher number than the default
fu <- follow_up(survey1, survey2, size_subset = 50, iterations = 10)
## Warning: There are some problems with the data.
## Warning: There are some problems with the new samples and the batches.
## Warning: There are some problems with the new data.
## Warning: There are some problems with the old data.

Following the previous new function it reports if there are problems with the observations. One can check each collection with check_data to know more about the problems found.

If you have already performed the experiment on your observations you can also check the distribution:

# Create the first batch
variables <- c("Sex", "Smoke", "Age")
survey1 <- survey1[, variables]
index1 <- design(survey1, size_subset = 50, iterations = 10)
## Warning: There might be some problems with the data use check_data().
r_survey <- inspect(index1, survey1)
# Create the second batch with "new" students
survey2 <- survey2[, variables]
survey2$batch <- NA
# Prepare the follow up
all_classroom <- rbind(r_survey, survey2)
fu2 <- follow_up2(all_classroom, size_subset = 50, iterations = 10)
## Warning: There are some problems with the data.
## Warning: There are some problems with the new samples and the batches.
## Warning: There are some problems with the new data.
## Warning: There are some problems with the old data.
tail(fu2)
## [1] "NewSubset2" "NewSubset2" "NewSubset2" "NewSubset2" "NewSubset2"
## [6] "NewSubset3"

Using this function will help to decide which new observations go to which new batches.

Closing remarks

The famous quote from Fisher goes:

“To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.”

This emphasizes the importance of involving a statistician early on in the experimental design process.
Unfortunately, in some cases, it may be too late to involve a statistician in the experimental design process or the reality of unforeseen circumstances messed the design of your carefully planned experiment.

My aim with this package is to provide practical tools for statisticians, bioinformaticians, and anyone who works with data. These tools are designed to be easy to use and can be used to analyze data in a variety of contexts. Let me know if it is helpful in your case.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.2.2 (2022-10-31)
##  os       Ubuntu 22.04.2 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language en_US
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2023-04-09
##  pandoc   2.19.2 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version  date (UTC) lib source
##  blogdown      1.16     2022-12-13 [1] CRAN (R 4.2.2)
##  bookdown      0.33     2023-03-06 [1] CRAN (R 4.2.2)
##  bslib         0.4.2    2022-12-16 [1] CRAN (R 4.2.2)
##  cachem        1.0.7    2023-02-24 [1] CRAN (R 4.2.2)
##  cli           3.6.1    2023-03-23 [1] CRAN (R 4.2.2)
##  digest        0.6.31   2022-12-11 [1] CRAN (R 4.2.2)
##  evaluate      0.20     2023-01-17 [1] CRAN (R 4.2.2)
##  experDesign * 0.2.0    2023-04-05 [1] CRAN (R 4.2.2)
##  fastmap       1.1.1    2023-02-24 [1] CRAN (R 4.2.2)
##  htmltools     0.5.4    2022-12-07 [1] CRAN (R 4.2.2)
##  jquerylib     0.1.4    2021-04-26 [1] CRAN (R 4.2.2)
##  jsonlite      1.8.4    2022-12-06 [1] CRAN (R 4.2.2)
##  knitr         1.42     2023-01-25 [1] CRAN (R 4.2.2)
##  MASS        * 7.3-58.1 2022-08-03 [2] CRAN (R 4.2.2)
##  R6            2.5.1    2021-08-19 [1] CRAN (R 4.2.2)
##  rlang         1.1.0    2023-03-14 [1] CRAN (R 4.2.2)
##  rmarkdown     2.20     2023-01-19 [1] CRAN (R 4.2.2)
##  rstudioapi    0.14     2022-08-22 [1] CRAN (R 4.2.2)
##  sass          0.4.5    2023-01-24 [1] CRAN (R 4.2.2)
##  sessioninfo   1.2.2    2021-12-06 [1] CRAN (R 4.2.2)
##  xfun          0.37     2023-01-31 [1] CRAN (R 4.2.2)
##  yaml          2.3.7    2023-01-23 [1] CRAN (R 4.2.2)
## 
##  [1] /home/lluis/bin/R/4.2.2
##  [2] /opt/R/4.2.2/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

rtweet future

Thu, 16 Feb 2023 00:00:00 +0000

Background: how I became the maintainer of rtweet

I didn’t want to maintain rtweet. It might sound strange coming from its maintainer, but I didn’t want the responsibility of writting software 12k people install it monthly. My offer was always to help with it so that the users could benefit from improvements and bug fixes on the package. I initially thought that having permissions to close issues, label them would help the community and the maintainer.

I was not totally altruistic. As recently recommended by Maëlle and Steffi, I had some interest in the package. I had a bot posting some plots daily. I wanted to have alt text in the tweets. There was a pending PR for just that. I could fork and use my own version or help with the package.

I got in touch with rOpenSci about the package. After some time waiting hearing back from the author of the package (Many thanks Michael W. Kearney, I was given edit permissions to the repository (under the helping eye of Scott Chamberlain). At the time, I got permission from rOpenSci there were 167 issues and PR open.

As I started going through them and changing the source code, people showed up to help (Thanks!). New contributors helped with closing issues via new PR or simply advising about naming functions or asking for its future. Another developer gained access to repository and contributed with their expertise (resulting in some funny moments). Despite all (breaking) changes, we tried to make the package easier to maintain. After waiting some time, for further community feedback, I released a new version through CRAN.

Shortly after, for some reasons my bot got less views and engagement. It made me realize that it might be an opportunity to start a professional project around the content of what this bot did. In addition, although I learn a lot from the online communities I decided to connect more with the local community. I am currently involved with re-launching the Barcelona R user group and organizing the Spanish R conference. These activities take time an energy away from package maintenance (not just rtweet). However, my support for the users of rtweet remains.

The community support I received was great. I hope to pass on it. CC-BY-4.0 Artwork by @allison_horst.

Current situation: supporting the v1 and adding support to the v2 API

To maintain rtweet I needed to adapt and understand the Twitter’s API (v1) documentation. The documentation didn’t always match the functionality. rtweet follows the API tightly, any change might break the package. For instance, the addition of the edit field broke the parsing of the tweets.

At the same time one of the oldest issues in rtweet is moving to the new API v2. With the 1.0 release, I decided it was time to stop adding features relying in it. It was clear that Twitter was moving to deprecate and replace it with the newer API. There are many benefits of the new API to developer and users. But as I realized that I should still fix bugs related to them.

The 1.1 release have recently provided some bug fixes and support for the v2. There is also an outstanding issue preventting new users of connecting to Twitter (fixed in github). The streaming endpoint in v1 stopped working before the announced date. So the new release included support for the replacement endpoint in v2.

As you have guessed, between releases I had been thinking and working to support API v2. The foundations to support the new endpoints were already set and easily expanded to new endpoints. In the development version of the package (in the devel branch), it is possible to retrieve bookmarks, retrieve the archive if you have academic access, among other endpoints.

Going forward: supporting rtweet users of v2 API

Recently (~3 weeks), there have been anouncements of future changes in who and how will be able to access the API. Simultaneously, there have been unannounced restrictions affecting other tools using the API. Recent changes and announcements are driven by need of money to sustain Twitter. To maintain rtweet with its current functionality it might need funds.

I will continue supporting rtweet and the freely accessible endpoints. But given that I will have less time and energy for rtweet, I am looking for a co-maintainer to help me:

Supporting new endpoints, using httr2, testing an API in CI, …
Review changes to avoid new bugs.
Help with issues and questions that the transition to API v2 and the current uncertainty bring.

Get in touch with me in this issue.

From time to time, I receive bug reports and petitions from premium API users. Currently, premium users get access to more data, elevated request rates and other endpoints. Helping them is usually challenging, and developing support for these endpoints will be difficult. To continue supporting these users I’ve set up a ☕ buymeacoffee and I, and my co-maintainer, will be open to consulting jobs about rtweet. With these jobs and funding we will be able to support them and implement new endpoints for all the users.

There is scarce information about the API changes and prices. Changes might come suddenly. We’ll do our best to keep up and inform all users. I hope this will be a good way to continue supporting the package an the community of users. Let me know if you have other suggestions.

Decorating/contributing to R. CC-BY-4.0 artwork by @allison_horst.

Thanks Maëlle for your support and reviewing the post.

Accessing REDCap from R

Wed, 08 Feb 2023 00:00:00 +0000

In this post, I want to summarize some of the packages to connect to REDCap. For those who don’t know, REDCap is a database designed for clinical usage, which allows easy data collection of patients’ responses by clinicians and interactions with the patients via surveys.

It has specific features such as scheduling surveys sent to patients, compatibility with tablets and mobile phones for data entry while visiting patients, grouping data in instruments (for repeating the same questions multiple times), multiple choice and check buttons, and different arms (like paths for patients). Most importantly is relatively easy to manage by clinical administrators.

In CRAN there are ~11 packages mentioning it at the time of writing it. The purpose of this post is to help decide which packages can be helpful in which situations. This post won’t be a deep analysis or comparison of capabilities, it describes some of the best and worse features of each package.

REDCapR

REDCapR is the official package to connect to the database. It allows you to read, write and filter the requests. It has some security-related functions.

REDCapTidieR

REDCapTidieR is a package that provides summaries of tables and helps with nested tibbles data by arm. It depends on REDCapR.

tidyREDCap

tidyREDCap is a package that simplifies the tables for instruments and choose-all or choose-one question types. It is easy to make tables and it depends on REDCapR. It requires the first and last columns to make instruments.

Screenshot of a design with several instruments in a single arm (from https://www.project-redcap.org/)

REDCapExporter

REDCapExporter is a package to build a data package from a database for redistribution. It does not depend on REDCapR.

redcapAPI

redcapAPI is a package for making data accessible and analysis-ready as quickly as possible with huge documentation in a wiki but has no vignette or examples and it does not depend on REDCapR.

REDCapDM

REDCapDM is a package that provides functions to read and manage REDCap data and identify missing or extreme values as well as transform the data provided by the API. It depends on REDCapR.

ReviewR

ReviewR is a package that creates a shiny website with data from the database to explore it. It uses the REDCapR to connect to your instance.

rccola

rccola is a package to provide a secure connection to the database but it doesn’t provide any handling of the data. It uses redcapAPI to connect to the database.

Other packages

Other packages mention REDCap:

nmadb: which implements its own connection procedure for a specific REDCap database of network meta-analyses.
distcomp: Allows to do computation on a distributed data also in REDCap.
cgmanalysis: which mentions that data produced is compatible with REDCap.

Conclusion

I’m sure that many packages briefly described here can do much more than what I understood from a glance at their documentation and DESCRIPTION.

Most packages provide some data for the examples (and probably tests), while others do not. This is a technical problem that might impact users if there are no examples in the functions.

REDCapR is used by most packages to access the database, but most of the packages focus on transforming the data provided by the API (or data exported) or the exported data. It highlights that the data exported is useful but that depending on the preferences of the users it needs to be transformed for easy usage.

Exploring CRAN's files: part 2

Thu, 28 Jul 2022 00:00:00 +0000

Introduction

In the first post of the series we briefly explored packages available on CRAN. Now I’ll focus on history of the packages and its size using the following files:

packages <- tools::CRAN_package_db()
current <- tools:::CRAN_current_db()
archive <- tools:::CRAN_archive_db()

In this part we will use two files: The current and the archive, let’s see why.

current file

The current database has has the package size, dates of modification, which I assume is date added to CRAN and user name of who last modified it. This is the same information returned by file.info

current[1, 1:10]
##     size isdir mode               mtime               ctime               atime
## A3 42810 FALSE  664 2015-08-16 23:05:54 2022-09-03 12:02:27 2022-09-03 14:00:19
##     uid  gid  uname    grname
## A3 1001 1001 hornik cranadmin

archive file

The archive database returns the same information, but as you might guess by the name it doesn’t provide information about current packages but for packages in the archive and no longer available by default.

archive[[1]]
##                     size isdir mode               mtime               ctime
## A3/A3_0.9.1.tar.gz 45252 FALSE  664 2013-02-07 10:00:29 2022-08-22 18:14:53
## A3/A3_0.9.2.tar.gz 45907 FALSE  664 2013-03-26 19:58:40 2022-08-22 18:14:53
##                                  atime  uid  gid  uname    grname
## A3/A3_0.9.1.tar.gz 2022-08-22 17:39:50 1001 1001 hornik cranadmin
## A3/A3_0.9.2.tar.gz 2022-08-22 17:39:50 1010 1001 ligges cranadmin

The date matches that available on the web’s old sources, so we can be confident of it’s meaning.

CRAN history

As we have seen there are some files about the archives of CRAN. These include information about date of modification (moving/editing) and user who did it and of course name and sometimes version of the package. These archives are the great treasure of CRAN because they help to make reproducible long time ago run experiments or analysis.

Note that I’m not totally sure that this archive contains the full record of packages, some initial packages might be missing. I’m also aware of some packages removed by CRAN which do not longer appear on this records.

Nevertheless, this should provide an accurate picture of packages available through time. Also as there is no information when a package is archived (here, there is on PACKAGES.in) so I might overestimate the packages available at any given moment.

Remember the plot about acceptance of packages on CRAN? That plot only looked at current packages available, let’s check it with all the archive:

Figure 1: Packages on CRAN archive by their addition to it. There are over 125000 archives on CRAN.

All these packages come from packages with few releases and packages with many releases. If we look at which packages had the most releases:

package	Releases
spatstat	206
Matrix	204
mgcv	162
RcppArmadillo	150
rgdal	146
nlme	143
caret	139
spdep	139
lattice	137
plotrix	131
sp	128
XML	126
Rcmdr	123
lme4	122
gstat	121
arm	119
foreign	117
party	117
maptools	113
raster	108

Surprisingly there are packages with more than 200 versions on CRAN!

Figure 2: Releases distirbution. Packages and number of releases

Most packages have 1 release, usually packages have 3, but the mean is around 6.

Given all this different versions of packages how big are all the packages on CRAN?

CRAN size

Have you ever wondered how big is CRAN? According to the memory size of the source packages all CRAN source packages are approximately 96.8 Gb.

This doesn’t include binaries for multiple architectures and OS. The package size might indicate whether the package has considerable amount of data.

Looking back to the size of the packages along time we can see this pattern:

Figure 3: Package and their median size. Archived packages have become bigger since 2014. Packages on CRAN have been getting bigger since 2017.

Packages available on CRAN are smaller than those no longer on CRAN. But versions of packages on CRAN that got archived are usually bigger than current versions. Packages no longer on CRAN are usually bigger. Median size of packages is increasing (quickly).

Figure 4: Size of package with releases. Package are usually small but seem to gain weight when updating.

Typically packages increase their size with each new release up to when they reach 50 releases. For higher releases this plot depends on very few packages and might not be representative.

Figure 5: Size of package with releases by availability. Packages no longer in CRAN are usually smaller than those in it. The continous black line is CRAN’s current threshold, while the discontinous black line is current median size.

Here we can appreciate better how packages tend to be below the CRAN threshold. There isn’t much of a difference between packages available on CRAN and those archived.

If we look at the size of package of the first release over time we’ll see a representative view:

Figure 6: Size of the first release by time. Package size increases with time with a peak around 2010 and increasing again since 2014 but still hasn’t surprased the previous record.

Package size tends to increase except for the brief period 2010-2014. Currently it increases less than before that period but is close to its maximum.

Conclusions

Most packages are not updated too much, between 1 and 3 times. But there are packages that are updated quite a lot, this might mean they are data packages and not software packages or that they have frequent minor and major updates.
Most current packages have smaller size than those archived. Packages no longer available usually had bigger size than those packages still on CRAN.
Surprisingly packages increase their size a lot till the 25 release. But also with time except for a period in 2010 and 2014. This decreasing period might be due to a change in CRAN policy.

Future parts

On future posts I’ll explore:

patterns accepting packages and updates in packages.
the relation between dependencies, initial release and updates.
who handled the packages.

Exploring CRAN's files: part 1

Sat, 23 Jul 2022 00:00:00 +0000

Introduction

There are many great things in base R, one of them is the tools package. This package has the functions that are used to build, check and create packages, documentation and manuals.

As I wanted to know how CRAN works and its changes I was looking into the source code of tools. I found some internal functions that access freely available files with information about CRAN packages. These private functions are at the CRANtools.R file.

packages <- tools::CRAN_package_db()
# current <- tools:::CRAN_current_db()
# archive <- tools:::CRAN_archive_db()
# issues <- tools::CRAN_check_issues()
# alias <- tools:::CRAN_aliases_db()
# rdxrefs <- tools:::CRAN_rdxrefs_db()

As I was not sure of the information on these files I asked on R-devel but I did not receive an answer. They seem to be quite obscure and as private functions might be removed without notice and shouldn’t be used in any dependency. However, as the files contain information about CRAN they might provide interesting clues about the history of CRAN and how it is operated.

On this post I will focus on the first file. I’ll explore a couple of fields and in future posts I will use the other files to explore more about CRAN history.

packages file

First of all a very brief exploration of what is in this file:

##    Package Version Priority                        Depends
## 1       A3   1.0.0     <NA> R (>= 2.15.0), xtable, pbapply
## 2 AATtools   0.0.1     <NA>                   R (>= 3.6.0)
## 3   ABACUS   1.0.0     <NA>                   R (>= 3.1.0)
##                                 Imports LinkingTo
## 1                                  <NA>      <NA>
## 2  magrittr, dplyr, doParallel, foreach      <NA>
## 3 ggplot2 (>= 3.1.0), shiny (>= 1.3.1),      <NA>
##                               Suggests Enhances    License License_is_FOSS
## 1                  randomForest, e1071     <NA> GPL (>= 2)            <NA>
## 2                                 <NA>     <NA>      GPL-3            <NA>
## 3 rmarkdown (>= 1.13), knitr (>= 1.22)     <NA>      GPL-3            <NA>

Packages has similar information as available.packages() but with many more columns with published date, reverse dependencies, X-CRAN-Comment, who packaged it… Also note that all this packages are not filtered to match R version, OS_type, subarch and there are almost duplicates (I learned about this filtering while reading the great documentation of available.packages() and also finding some mentions online).

As we have data from several years I’ll sometimes show the release dates of different R versions to provide some context. Without further delay let’s explore the data!

Published packages

CRAN started some time ago (in 1997) but it hasn’t remained frozen. The package archive (the A in CRAN) has been updating since then. For instance the current packages do not include packages that were removed, archived or those replaced by updates.

First packages are submitted to CRAN and once accepted they are published. As accepted and published usually are almost instantaneous I might use them as synonyms. Looking at the current available packages and their publication date, we can see the following:

Figure 1: Packages accepted on CRAN by the publication date.

The oldest package added was in 2010. This means a package without issues, dependencies changes, bugs detected by the automatic checks since 12 years!

The daily rate of acceptance has increased from less than 10 a day till 2020 to more than 30 this year 2022. If we summarize that information for month we see the same, but the little bump in 2020 disappears but we see other patterns:

Figure 2: Monthly packages published to CRAN. Some monthly variance is observed.

Instead of just one bump we see some waves with less packages on CRAN accepted late in the year and an increase of packages the first months of the year.

If we look at the accumulated packages on CRAN we see an exponential growth:

Figure 3: Acumulation of packages. Most of the packages have been published in the last 2 years.

In fact, most packages currently on CRAN where added since March 2021 than all the previous years.

Figure 4: Percentage of current packages on CRAN according to their date of publication. Most of them were published/updated on the last year and a half.

This is a good time to remind that the date being used is the date of publication of this version of the packages. Many had previous versions on CRAN:

First release	Packages
No	14,294
Yes	4,113

Processing time

Previously I found that CRAN submissions present some key differences between new packages and already published packages which impact how long do they need to wait to be published on CRAN. With the existing data we can compare how fast is the process by comparing the published date with the build date.

The build date is added to the tar.gz file automatically when the developer builds the package via R CMD build. However, the published date is set by CRAN once the packages are accepted on CRAN.

To visualize the differences I will also compare if there is some difference with new packages and those that were already on CRAN:

Figure 5: Histogram of time difference between building and publishing a package. Color indicates if the package is new to CRAN or not. Most of the published packages take more or less the same time regardless of if it is the first time or not.

There doesn’t seem to be much difference between date of building and date of publication according to if it is the first release or not. The precision is just a day and this is usually a fast process well below 50 days. Few packages exceed spend so much after build before publication and they are too few to be noticeable at this scale. Since 2016/05/02 there is a check that raises an issue if the build is older than a month.

Note that one might need to build multiple times the package before it is accepted. Packages published for the first time on CRAN might have been submitted previously, but when they finally built and pass the checks and manual review they are handled as fast as packages already on CRAN.

However, this time between build and acceptance might have changed with time:

Figure 6: Processing time between building the package and being published by date. There is a high difference between new packages and old ones. New packages usually take more time while existing packages take less than a day currently.

We clearly see a difference in processing time for those packages already on CRAN and those that are not. Keep in mind that for the few packages from before 2016 the estimation might not be accurate. At the same time this is consistent with the manual review process (For more information see my previous post about the review process of CRAN or my talk at the useR2021). It also means that there is a huge variation of time about how packages are handled. However this seems to be reducing: while in 2010 it took around 2 weeks, nowadays it takes less than a week and getting closer to a 1 day of median time between a package being built and appearing on CRAN that takes for existing packages.

This difference might be explainable due to experience: authors and maintainers whose package(s) are already in CRAN know better how to submit a new version without problems the checks.

It could also be that new packages need more time from the CRAN team. In 2020 we see it took longer than in previous years for packages to be added on CRAN. Maybe the increase in the processing time in 2020 was due the huge volume of submissions CRAN received or more checks on the developer side before submitting it to CRAN.

Both explanations are not mutually exclusive.

More packages published the same day mean more processing time? It doesn’t look like it.

Figure 7: Packages accepted the same day and processing time.New packages are accepted sooner than packages on CRAN respect to the builddate.

Surprisingly, we see a lot of variation on the delay of packages already accepted on CRAN. In addition, the more new packages accepted the same day, the less delay there is. I think this just means that when reviewers work on the submission queue several packages might be approved.

This might also mean packages have already been built several times before finally being accepted and now the errors, warnings and notes have been solved. Last, this could indicate that developers with their package already on CRAN wait a bit between building and submitting the package as the developer might be taking some time to double check before submission (dependencies, on several machines, other?) or a time zone difference (submitting in the noon of a region but at the reviewers night).

Conclusion

There are packages that for 12 years have been working without problems despite the several major changes in R (See figure 1). This speaks volumes of the packages’ quality, and the backward compatibility that the R core aims and CRAN checks.
CRAN accepts an incredible amount of packages daily and monthly. The system and the team are doing an incredible work mostly on their free time (See figure 2). Many thanks!
Accepted packages are handled very fast, in less than a week usually (See figure 7). But it is not possible to distinguish alone time in the submission system and time on the developer computer.

Future parts

We’ve explored a snapshot of current packages and a brief window of all the history of CRAN. There is much more that can be done with all the other files.

On future posts I’ll explore:

patterns accepting packages and updates in packages.
who handled the packages.
Size of packages.
the relation between dependencies, initial release and updates.

Other suggestions?

Edit: Many thanks to Maëlle Salmon and Dirk Eddelbuettel for their feedback on an initial version of this series of posts.

Reproducibility

## - Session info -------------------------------------------------------------------------------------------------------
##  setting  value
##  version  R version 4.2.1 (2022-06-23)
##  os       Ubuntu 20.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  C
##  ctype    C
##  tz       Europe/Madrid
##  date     2022-07-23
##  pandoc   2.18 @ /usr/lib/rstudio/bin/quarto/bin/tools/ (via rmarkdown)
## 
## - Packages -----------------------------------------------------------------------------------------------------------
##  package      * version    date (UTC) lib source
##  assertthat     0.2.1      2019-03-21 [2] RSPM (R 4.2.0)
##  base64enc      0.1-3      2015-07-28 [2] CRAN (R 4.0.0)
##  blogdown       1.10       2022-05-10 [2] RSPM (R 4.2.0)
##  bookdown       0.27       2022-06-14 [2] RSPM (R 4.2.0)
##  bslib          0.4.0      2022-07-16 [2] RSPM (R 4.2.0)
##  cachem         1.0.6      2021-08-19 [2] RSPM (R 4.2.0)
##  cli            3.3.0      2022-04-25 [2] RSPM (R 4.2.0)
##  codetools      0.2-18     2020-11-04 [2] RSPM (R 4.2.0)
##  colorspace     2.0-3      2022-02-21 [2] RSPM (R 4.2.0)
##  crayon         1.5.1      2022-03-26 [2] RSPM (R 4.2.0)
##  curl           4.3.2      2021-06-23 [2] RSPM (R 4.2.0)
##  data.table     1.14.2     2021-09-27 [2] RSPM (R 4.2.0)
##  DBI            1.1.3      2022-06-18 [2] RSPM (R 4.2.0)
##  digest         0.6.29     2021-12-01 [2] RSPM (R 4.2.0)
##  dplyr        * 1.0.9      2022-04-28 [2] RSPM (R 4.2.0)
##  ellipsis       0.3.2      2021-04-29 [2] RSPM (R 4.2.0)
##  evaluate       0.15       2022-02-18 [2] RSPM (R 4.2.0)
##  fansi          1.0.3      2022-03-24 [2] RSPM (R 4.2.0)
##  farver         2.1.1      2022-07-06 [2] RSPM (R 4.2.0)
##  fastmap        1.1.0      2021-01-25 [2] RSPM (R 4.2.0)
##  flextable    * 0.7.2      2022-06-12 [2] RSPM (R 4.2.0)
##  forcats      * 0.5.1      2021-01-27 [2] RSPM (R 4.2.0)
##  gdtools        0.2.4      2022-02-14 [2] RSPM (R 4.2.0)
##  generics       0.1.3      2022-07-05 [2] RSPM (R 4.2.0)
##  geomtextpath * 0.1.0      2022-01-24 [2] CRAN (R 4.2.1)
##  ggplot2      * 3.3.6.9000 2022-06-29 [2] Github (tidyverse/ggplot2@7571122)
##  ggrepel      * 0.9.1      2021-01-15 [2] RSPM (R 4.2.0)
##  glue           1.6.2      2022-02-24 [2] RSPM (R 4.2.0)
##  gtable         0.3.0      2019-03-25 [2] CRAN (R 4.0.0)
##  highr          0.9        2021-04-16 [2] RSPM (R 4.2.0)
##  htmltools      0.5.3      2022-07-18 [2] RSPM (R 4.2.0)
##  jquerylib      0.1.4      2021-04-26 [2] RSPM (R 4.2.0)
##  jsonlite       1.8.0      2022-02-22 [2] RSPM (R 4.2.0)
##  knitr          1.39       2022-04-26 [2] RSPM (R 4.2.0)
##  labeling       0.4.2      2020-10-20 [2] RSPM (R 4.2.0)
##  lattice        0.20-45    2021-09-22 [3] CRAN (R 4.2.0)
##  lifecycle      1.0.1      2021-09-24 [2] RSPM (R 4.2.0)
##  lubridate    * 1.8.0      2021-10-07 [2] RSPM (R 4.2.0)
##  magrittr       2.0.3      2022-03-30 [2] RSPM (R 4.2.0)
##  Matrix         1.4-1      2022-03-23 [2] RSPM (R 4.2.0)
##  mgcv           1.8-40     2022-03-29 [2] RSPM (R 4.2.0)
##  munsell        0.5.0      2018-06-12 [2] RSPM (R 4.2.0)
##  nlme           3.1-158    2022-06-15 [2] RSPM (R 4.2.0)
##  officer        0.4.3      2022-06-12 [2] RSPM (R 4.2.0)
##  pillar         1.8.0      2022-07-18 [2] RSPM (R 4.2.0)
##  pkgconfig      2.0.3      2019-09-22 [2] RSPM (R 4.2.0)
##  purrr          0.3.4      2020-04-17 [2] RSPM (R 4.2.0)
##  R6             2.5.1      2021-08-19 [2] RSPM (R 4.2.0)
##  Rcpp           1.0.9      2022-07-08 [2] RSPM (R 4.2.0)
##  rlang          1.0.4      2022-07-12 [2] RSPM (R 4.2.0)
##  rmarkdown      2.14       2022-04-25 [2] RSPM (R 4.2.0)
##  rstudioapi     0.13       2020-11-12 [2] RSPM (R 4.2.0)
##  rversions    * 2.1.1      2021-05-31 [2] RSPM (R 4.2.0)
##  sass           0.4.2      2022-07-16 [2] RSPM (R 4.2.0)
##  scales         1.2.0      2022-04-13 [2] RSPM (R 4.2.0)
##  sessioninfo    1.2.2      2021-12-06 [2] RSPM (R 4.2.0)
##  stringi        1.7.8      2022-07-11 [2] RSPM (R 4.2.0)
##  stringr        1.4.0      2019-02-10 [2] RSPM (R 4.2.0)
##  systemfonts    1.0.4      2022-02-11 [2] RSPM (R 4.2.0)
##  textshaping    0.3.6      2021-10-13 [2] RSPM (R 4.2.0)
##  tibble         3.1.7      2022-05-03 [2] RSPM (R 4.2.0)
##  tidyr        * 1.2.0      2022-02-01 [2] RSPM (R 4.2.0)
##  tidyselect     1.1.2      2022-02-21 [2] RSPM (R 4.2.0)
##  utf8           1.2.2      2021-07-24 [2] RSPM (R 4.2.0)
##  uuid           1.1-0      2022-04-19 [2] RSPM (R 4.2.0)
##  vctrs          0.4.1      2022-04-13 [2] RSPM (R 4.2.0)
##  withr          2.5.0      2022-03-03 [2] RSPM (R 4.2.0)
##  xfun           0.31       2022-05-10 [2] RSPM (R 4.2.0)
##  xml2           1.3.3      2021-11-30 [2] RSPM (R 4.2.0)
##  yaml           2.3.5      2022-02-21 [2] RSPM (R 4.2.0)
##  zip            2.2.0      2021-05-31 [2] RSPM (R 4.2.0)
## 
##  [1] /home/lluis/bin/R/4.2.1
##  [2] /usr/lib/R/site-library
##  [3] /usr/lib/R/library
## 
## ----------------------------------------------------------------------------------------------------------------------

Upgrading rtweet to 1.0.2

Mon, 04 Jul 2022 00:00:00 +0000

In this post I will provide some examples of what has changed between rtweet 0.7.0 and rtweet 1.0.2. I hope both the changes and this guide will help all users. I highlight the most important and interesting changes in this blog post, and for a full list of changes you can consult it on the NEWS.

Big breaking changes

More consistent output

This is probably what will affect the most users. All functions that return data about tweets¹ now return the same columns.

For example if we search some tweets we’ll get the following columns:

> tweets <- search_tweets("weather")
> colnames(tweets)
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "full_text"                    
 [5] "truncated"                     "display_text_range"           
 [7] "entities"                      "metadata"                     
 [9] "source"                        "in_reply_to_status_id"        
[11] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[13] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[15] "geo"                           "coordinates"                  
[17] "place"                         "contributors"                 
[19] "is_quote_status"               "retweet_count"                
[21] "favorite_count"                "favorited"                    
[23] "retweeted"                     "lang"                         
[25] "quoted_status_id"              "quoted_status_id_str"         
[27] "quoted_status"                 "possibly_sensitive"           
[29] "retweeted_status"              "text"                         
[31] "favorited_by"                  "scopes"                       
[33] "display_text_width"            "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "query"                         "withheld_scope"               
[41] "withheld_copyright"            "withheld_in_countries"        
[43] "possibly_sensitive_appealable"

rtweet now minimizes the processing of tweets and only returns the same data as provided by the API while making it easier to handle by R. However, to preserve the nested nature of the data returned some fields are now nested inside other. For example, previously fields "bbpx_coords", "geo_coords", "coords_coords" were returned as separate columns, but they are now nested inside "place", "coordinates" or "geo" depending where they are provided. Some columns previously calculated by rtweet are now not returned, like "rtweet_favorite_count". At the same time it provides with new columns about each tweet like the "withheld_*" columns.

If you scanned through the columns you might have noticed that columns "user_id" and "screen_name" are no longer returned. This data is still returned by the API but it is now made available to the rtweet users via users_data():

> colnames(users_data(tweets))
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

This blog post should help you find the right data columns, but if you don’t find what you are looking for it might be nested inside a column.
Try using dplyr::glimpse() to explore the data and locate nested columns. For example the entities column (which is present in both tweets and users) have the following useful columns:

> names(tweets$entities[[1]])
[1] "hashtags"      "symbols"       "user_mentions" "urls"         
[5] "media"

Similarly if you look up a user via search_users() or lookup_users() you’ll get consistent data:

> users <- lookup_users(c("twitter", "rladiesglobal", "_R_Foundation"))
> colnames(users)
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

You can use tweets_data() to retrieve information about their latest tweet:

> colnames(tweets_data(users))
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "text"                         
 [5] "truncated"                     "entities"                     
 [7] "source"                        "in_reply_to_status_id"        
 [9] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[11] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[13] "geo"                           "coordinates"                  
[15] "place"                         "contributors"                 
[17] "is_quote_status"               "retweet_count"                
[19] "favorite_count"                "favorited"                    
[21] "retweeted"                     "lang"                         
[23] "retweeted_status"              "possibly_sensitive"           
[25] "quoted_status"                 "display_text_width"           
[27] "user"                          "full_text"                    
[29] "favorited_by"                  "scopes"                       
[31] "display_text_range"            "quoted_status_id"             
[33] "quoted_status_id_str"          "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "metadata"                      "query"                        
[41] "withheld_scope"                "withheld_copyright"           
[43] "withheld_in_countries"         "possibly_sensitive_appealable"

You can merge them via:

users_and_last_tweets <- cbind(users, id_str = tweets_data(users)[, "id_str"])

In the future (see below), with helper functions managing the output of rtweet will become easier.

Finally, get_followers() and get_friends() now return the same columns:

> colnames(get_followers("_R_Foundation"))
[1] "from_id" "to_id"  
> colnames(get_friends("_R_Foundation"))
[1] "from_id" "to_id"

This will make it easier to build networks of connections (although you might want to convert screen names to ids or vice versa).

More consistent interface

All paginated functions that don’t return tweets now use a consistent pagination interface (except the premium endpoints). They all store the “next cursor” in an rtweet_cursor attribute, which will be automatically retrieved when you use the cursor argument. This will make it easier to continue a query you started:

users <- get_followers("_R_Foundation")
users
     
# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

They support max_id and since_id to find earlier and later tweets respectively:

# Retrieve all the tweets made since the previous request
newer <- search_tweets("weather", since_id = tweets)
# Retrieve tweets made before the previous request
older <- search_tweets("weather", max_id = tweets)

If you want more tweets than it is allowed by the rate limits of the API, you can use retryonratelimit to wait as long as needed:

long <- search_tweets("weather", n = 1000, retryonratelimit = TRUE)

This will keep busy your terminal until the 1000 tweets are retrieved.

Saving data

An unexpected consequence of returning more data (now matching that returned by the API) is that it is harder to save it in a tabular format. For instance one tweet might have one media, mention two users and have three hashtags. There isn’t a simple way to save it in a single row uniformly for all tweets or it could lead to confusion.

This resulted in deprecating save_as_csv, read_twitter_csv and related functions because they don’t work with the new data structure and it won’t be possible to load the complete data from a csv. They will be removed in later versions.

Many users will benefit from saving to RDS (e.g., saveRDS() or readr::write_rds()), and those wanting to export to tabular format can simplify the data to include only that of interest before saving with generic R functions (e.g., write.csv() or readr::write_csv()).

Other breaking changes

Accessibility is important and for this reason if you tweet via post_tweet() and add an image, gif or video you’ll need to provide the media alternative text. Without media_alt_text it will not allow you to post.
tweet_shot() has been deprecated as it no longer works correctly. It might be possible to bring it back, but the code is complex and I do not understand enough to maintain it. If you’re interested in seeing this feature return, checkout the discussion about this issue and let me know if you have any suggestions.
rtweet also used to provide functions for data on emojis, langs and stopwordslangs. These are useful resources for text mining in general - not only in tweets - however they need to be updated to be helpful and would be better placed in other packages, for instance emojis is now on the bdpar package. Therefore they are no longer available in rtweet.
The functions like suggested_*() have been removed as they have been broken since 2019.

Easier authentication

An exciting part of this release has been a big rewrite of the authentication protocol. While it is compatible with previous rtweet authentication methods it has also some important new functions which make it easier to work with rtweet and the twitter API in different ways.

Different ways to authenticate

If you just want to test the package, use the default authentication auth_setup_default() that comes with rtweet. If you use it for one or two days you won’t notice any problem.

If you want to use the package for more than a couple of days, I recommend you set up your own token via rtweet_user(). It will open a window to authenticate via the authenticated account in your default browser. This authentication won’t allow you to do everything but it will avoid running out of requests and being rate-limited.

If you plan to make heavy use of the package, I recommend registering yourself as developer and using one of the following two mechanisms, depending on your plans:

Collect data and analyze: rtweet_app().
Set up a bot: rtweet_bot()

Find more information in the Authentication with rtweet vignette.

Storing credentials

Previously rtweet saved each token created, but now non-default tokens are only saved if you ask. You can save them manually via auth_save(token, "my_app"). Bonus, if you name your token as default (auth_save(token, "default")) it will be used automatically upon loading the library.

Further, tokens are now saved in the location output by tools::R_user_dir("rtweet", "config"), rather than in your home directory. If you have previous tokens saved or problems identifying which token is which use auth_sitrep(). This will provides clues to which tokens might be duplicated or misconfigured but it won’t check if they work. It will also automatically move your tokens to the new path.

To check which credentials you have stored use auth_list() and load them via auth_as("my_app"). All the rtweet functions will use the latest token loaded with auth_as (unless you manually specify one when calling it). If you are not sure which token you are using you can use auth_get() it will return the token in use, list them or ask you to authenticate.

Other changes of note

This is a list of other changes that aren’t too big or are not breaking changes but are worthy enough of a mention:

Iteration and continuation of requests

Using cursors, pagination or waiting until you can make more queries is now easier. For example you can continue previous requests via:

users <- get_followers("_R_Foundation")
users

# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

Additions

There is now a function to find a thread of a user. You can start from any tweet and it will find all the tweets of the thread: tweet_threading("1461776330584956929").

There is a lot of interest in downloading and keeping track of interactions on Twitter. The amount of interest is big enough that Twitter is releasing a new API to provide more information of this nature.

Future

Twitter API v2 is being released and soon it will replace API v1. rtweet up to now, including this release, uses API v1 so it will need to adapt to the new endpoints and new data returned.

First will be the streaming endpoints in November, so expect more (breaking?) changes around those dates if not earlier.

I would also like to make it easier for users, dependencies and the package itself to handle the outputs. To this regard I would like to provide some classes to handle the different type of objects it returns.

This will help avoid some of the current shortcomings. Specifically I would like to provide functions to make it easier to reply to previous tweets, extract nested data, and subset tweets and the accompanying user information.

Conclusions

While I made many breaking changes I hope these changes will smooth future development and help both users and maintainers.

Feel free to ask on the rOpenSci community if you have questions about the transition or find something amiss. Please let me know! It will help me prioritize which endpoints are more relevant to the community. (And yes, the academic archive endpoint is on the radar.)

It is also possible that I overlooked something and I thought the code is working when it isn’t. For example, after several months of changing the way the API is parsed, several users found it wasn’t handling some elements. Let me know of such or similar cases and I’ll try to fix it.

In case you find a bug, check the open issues and if it has not already been reported, open an issue on GitHub. Don’t forget to make a reprex and if possible provide the id of the tweets you are having trouble with. Unfortunately it has happened that when I came to look at a bug I couldn’t reproduce it as I wasn’t able to find the tweet which caused the error.

This release includes contributions from Hadely Wicham, Bob Rudis, Alex Hayes, Simon Heß, Diego Hernán, Michael Chirico, Jonathan Sidi, Jon Harmon, Andrew Fraser and many other that reported bugs or provided feedback. Many thanks all for using it, your interest to keep it working and improving rtweet for all.

Finally, you can read the whole NEWS online and the examples.

Happy tweeting!

Acknowledgements

This is a repost of the entry for rOpenSci. The post was edited and improved by Yanina Bellini Saibene and Steffi LaZerte, the community manager and assistant. Many thanks

Specifically these: get_favorites(), get_favorites_user(), get_mentions(), get_my_timeline(), get_retweets(), get_timeline(), get_timeline_user(), lists_statuses(), lookup_statuses(), lookup_tweets(), search_30day(), search_fullarchive(), search_tweets(), tweet_shot() and tweet_threading().↩︎

Reasons why packages are archived on CRAN

Tue, 07 Dec 2021 00:00:00 +0000

On the Repositories working group of the R Consortium Rich FitzJohn posted a comment to a file that seems to be were the CRAN team stores and uses to check the package history.

The structure is not defined anywhere I could find (I haven’t looked much to be honest).

Package: <package name>
X-CRAN-Comment: Archived on YYYY-MM-DD as <reason>.
X-CRAN-History: Archived on YYYY-MM-DD as <reason>.
  Unarchived on YYYY-MM-DD.
  .
  <Optional clarification of archival reason>
<Optional fields like License_restricts_use, Replaced_by, Maintainer: ORPHANED, OS_type: unix>

I think the X-CRAN-Comment is what appears on the website of an archived package, like on radix package. However, other comments on the website do not appear on that file.

In addition, the file doesn’t have some records of archiving and unarchiving of some packages, but there are old records from 2013 or before to now. But we can use it to see understand what are the reasons of archiving packages, which seems to be the main purpose of the file.

The data

First step is read the record. As it seems that it has some key: value structure similar to DESCRIPTION file of packages it seems it is a DCF format: Debian Control File format which is easy to read with the read.dcf function.

Exploring

A brief exploration of the data:

comment	history	packages
yes	no	3612
no	yes	2345
yes	yes	434
no	no	70

Many packages have either comments or history but relatively few both. I’m not sure when either of them is used, as I would expect that all that have history would have a comment.

Replaced_by	packages
no	6360
yes	101

Many packages are simply replaced by some other package.

Maintainer	packages
no	6366
yes	95

Most of the packages that have a Maintainer field are orphaned/archived. Does it mean that all the others are not orphaned?

Extracting reasons

Now that it is in R data structure, we can extract the relevant information, dates, type of action and reasons for each archivation event. I use strcapture for this task with a regex to extract the action, the date and the explanation it migh have.

I don’t know how the file is written probably it is a mix of automated tools and manual editing so there isn’t a simple way to collect all the information in a structured way. Simply because the structure has been changing along the years as well as the details of what is stored has changed, or there are missing events. However, the extracted information should be enough for our purposes.

Action	Events
archived	7096
orphaned	341
removed	113
renamed	2
replaced	4
unarchived	2973

As expected the most common recorded event are archivations, but there are some orphaned packages and even some removed packages. Also note the number of orphaned packages is greater than those with the Maintainer field, supporting my theory that the format has changed and that this shouldn’t be taken as an exhaustive and complete analysis of archivations.

How are they along time?

Even if there are some events recorded from 2009 it seems that this file has been more used more recently (last commit related to this was on 2015). I know that there are some old events not recorded on the file, because there are some packages currently present on CRAN that they had been archived but do not have an unarchived action, so conversely it could happen. So, this doesn’t necessarily mean that there are currently more packages archived from CRAN. But it is a clear indication that now at least there is a more accurate record of archived packages on this file.

Another source of records of archived packages might be cranberries. It would be nice to compare this file with the records on the database there.

Now that most of the package events are collected and we have the reasons of the actions, we can explore and classify the reasons. Using some simple regex I explore for key words or sentences.

We can look at the most frequent error reasons for archiving packages, patterns I found with more than 100 cases:

The most frequent error is that errors are not corrected or checks, even when there are reminders.
Next are the packages archived because they depend on other packages already not on CRAN.
There are some packages that are replaced by others and some maintainers might not want to continue supporting the package when they receive a message from CRAN about fixing an error.

Policy violation makes to the top 5 but with less than 500 events. Dependencies problems are the sixth cause, followed by email errors (bouncing, incorrect email…) and then come very sporadic problems about license, not fixing on updates of R, authorship problems or requests from authors.

Some of these errors happen at the same time for each event, but grouping these reasons together we get a similar table:

package_not_corrected	request_maintainer	dependencies	other	events
yes	no	no	no	4366
no	no	no	no	1530
no	no	yes	no	767
no	no	no	yes	374
yes	no	no	yes	15
yes	no	yes	no	13
no	no	yes	yes	2
yes	no	yes	yes	2
yes	yes	no	no	2
yes	yes	no	yes	1

Surprisingly the second most frequent group of archiving actions are due to many different reasons. This is probably the Pareto’s principle in action because they are around 15% of the archiving events but the causes are very diverse between them.

However, if we look at the packages which were archived (not at the request of maintainers), most of them just happen once:

Events	packages
1	5304
2	594
3	115
4	31
5	8

This suggests that once a package is archived maintainers do not make the effort to put it back on CRAN except on very few cases were there are multiple attempts. To check we can see the current available packages and see how many of those are still present on CRAN:

CRAN	Packages	Proportion
no	3869	64%
yes	2183	36%

Many packages are currently on CRAN despite their past archivation but close to 64% are currently not on CRAN.

Almost all that are on CRAN have now no X-CRAN-Comment, except for a few:

Package	X-CRAN-Comment
geiger	Orphaned and corrected on 2022-05-09. Repeated notifications about USE_FC_LEN_T were ignored.
alphahull	Versions up to 2.3 have been removed for mirepresentation of authorship.
udunits2	Orphaned on 2022-01-06 as installation problems were not corrected.
bibtex	Orphaned and corrected on 2020-09-19 as check problems were not corrected in time.

CRAN team might have missed these few packages and didn’t move the comments to X-CRAN-history.

There are some packages that are not archived that don’t have a CRAN-history happens too, but they usually have other fields changed.

Discussion

Most packages archived on CRAN are due to the maintainers not correcting errors found on the package by CRAN checks. It is clear that the checks that CRAN help packages to have a high quality but it has high cost on the maintainers and specially on CRAN team. Maintainers don’t seem to have enough time to fix the issues on time. And the CRAN team sends personalized reminders to maintainers and sometimes patches to the packages.

Although the desire to have packages corrected and with no issues is the common goal there are few options on light of these:

Be more restrictive

Prevent a package to be accepted if it breaks dependencies or archive packages when they fail checks. This will make it harder to keep packages on CRAN but would lift some pressure on the CRAN team. This would go against the current on other languages repositories, which often they don’t check the packages/modules and even have less restrictions on dependencies (so it might be an unpopular decision).
Be more permissive:

One option would be to allow for more time for maintainers to fix issues. I haven’t find any report of how long does it take for a package since an error to a fix on CRAN but often it is quite long. I have seen packages with a warning for months if not years and they weren’t archived from CRAN.

Maybe if users get a warning on installing packages that a package or one of its dependencies is not clear on all CRAN checks (without error or warnings). This might help to make users more conscious of their dependencies but this might add pressure to maintainers who already don’t have enough time to fix the problems of their packages.
Provide more help or tools to maintainers

Another option is to provide a mechanism for maintainers to receive help or fix the package. Currently CRAN requires that new packages that break dependencies to give enough notice in advance to other maintainers to fix their package. On R-pkg-devel mailing list there are often requests for help on submitting and fixing some errors detected by CRAN checks which often result on other maintainers sharing their solutions for the same problem.

There high percentage of packages that once archived do not come back to CRAN might be a good place to start helping maintainers and an opportunity for users to step in and help maintainers of packages they have been using. There is need for something else? How would that work?

At the same time it is admirable that after so many years there are few errors on the data. However, the archival process might be a good process to automate, providing the reason on the webpage and add it to X-CRAN-Comment and moving the comments to X-CRAN-History once it is unarchived. Knowing more about how these actions are performed by the CRAN team and how the community could help on the process will be beneficial to all.

Note: This blog was updated on 2022/01/02 to improve the parsing of actions and dates on packages. Resulting on a change on the first plot to include unarchived which slightly modified the second plot of reasons why packages are archived. This overall only affected the numbers of the plots not the conclusions or discussion.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.2.0 (2022-04-22)
##  os       Ubuntu 20.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2022-05-09
##  pandoc   2.17.1.1 @ /usr/lib/rstudio/bin/quarto/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package      * version date (UTC) lib source
##  assertthat     0.2.1   2019-03-21 [1] CRAN (R 4.2.0)
##  blogdown       1.9     2022-03-28 [1] CRAN (R 4.2.0)
##  bookdown       0.26    2022-04-15 [1] CRAN (R 4.2.0)
##  bslib          0.3.1   2021-10-06 [1] CRAN (R 4.2.0)
##  cli            3.3.0   2022-04-25 [1] CRAN (R 4.2.0)
##  colorspace     2.0-3   2022-02-21 [1] CRAN (R 4.2.0)
##  ComplexUpset * 1.3.3   2021-12-11 [1] CRAN (R 4.2.0)
##  crayon         1.5.1   2022-03-26 [1] CRAN (R 4.2.0)
##  DBI            1.1.2   2021-12-20 [1] CRAN (R 4.2.0)
##  digest         0.6.29  2021-12-01 [1] CRAN (R 4.2.0)
##  dplyr        * 1.0.9   2022-04-28 [1] CRAN (R 4.2.0)
##  ellipsis       0.3.2   2021-04-29 [1] CRAN (R 4.2.0)
##  evaluate       0.15    2022-02-18 [1] CRAN (R 4.2.0)
##  fansi          1.0.3   2022-03-24 [1] CRAN (R 4.2.0)
##  farver         2.1.0   2021-02-28 [1] CRAN (R 4.2.0)
##  fastmap        1.1.0   2021-01-25 [1] CRAN (R 4.2.0)
##  generics       0.1.2   2022-01-31 [1] CRAN (R 4.2.0)
##  ggplot2      * 3.3.6   2022-05-03 [1] CRAN (R 4.2.0)
##  glue           1.6.2   2022-02-24 [1] CRAN (R 4.2.0)
##  gtable         0.3.0   2019-03-25 [1] CRAN (R 4.2.0)
##  highr          0.9     2021-04-16 [1] CRAN (R 4.2.0)
##  htmltools      0.5.2   2021-08-25 [1] CRAN (R 4.2.0)
##  jquerylib      0.1.4   2021-04-26 [1] CRAN (R 4.2.0)
##  jsonlite       1.8.0   2022-02-22 [1] CRAN (R 4.2.0)
##  knitr          1.39    2022-04-26 [1] CRAN (R 4.2.0)
##  labeling       0.4.2   2020-10-20 [1] CRAN (R 4.2.0)
##  lifecycle      1.0.1   2021-09-24 [1] CRAN (R 4.2.0)
##  magrittr       2.0.3   2022-03-30 [1] CRAN (R 4.2.0)
##  munsell        0.5.0   2018-06-12 [1] CRAN (R 4.2.0)
##  patchwork      1.1.1   2020-12-17 [1] CRAN (R 4.2.0)
##  pillar         1.7.0   2022-02-01 [1] CRAN (R 4.2.0)
##  pkgconfig      2.0.3   2019-09-22 [1] CRAN (R 4.2.0)
##  purrr          0.3.4   2020-04-17 [1] CRAN (R 4.2.0)
##  R6             2.5.1   2021-08-19 [1] CRAN (R 4.2.0)
##  rlang          1.0.2   2022-03-04 [1] CRAN (R 4.2.0)
##  rmarkdown      2.14    2022-04-25 [1] CRAN (R 4.2.0)
##  rstudioapi     0.13    2020-11-12 [1] CRAN (R 4.2.0)
##  sass           0.4.1   2022-03-23 [1] CRAN (R 4.2.0)
##  scales         1.2.0   2022-04-13 [1] CRAN (R 4.2.0)
##  sessioninfo    1.2.2   2021-12-06 [1] CRAN (R 4.2.0)
##  stringi        1.7.6   2021-11-29 [1] CRAN (R 4.2.0)
##  stringr        1.4.0   2019-02-10 [1] CRAN (R 4.2.0)
##  tibble         3.1.7   2022-05-03 [1] CRAN (R 4.2.0)
##  tidyselect     1.1.2   2022-02-21 [1] CRAN (R 4.2.0)
##  utf8           1.2.2   2021-07-24 [1] CRAN (R 4.2.0)
##  vctrs          0.4.1   2022-04-13 [1] CRAN (R 4.2.0)
##  withr          2.5.0   2022-03-03 [1] CRAN (R 4.2.0)
##  xfun           0.30    2022-03-02 [1] CRAN (R 4.2.0)
##  yaml           2.3.5   2022-02-21 [1] CRAN (R 4.2.0)
## 
##  [1] /home/lluis/bin/R/4.2.0/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Workshop: Creating packages

Wed, 20 Oct 2021 00:00:00 +0000

I was invited by Francis Mensha on R-devel slack to give this workshop on April 29th for the R User Group Ghana.

First time teaching R stats and remote I expected people to follow the workshop and run the code and then ask questions but I was asked to do it myself. Hope it was easy enough to follow.