reviews | B101nfo

CRAN review

Sun, 31 Jan 2021 00:00:00 +0000

I’ve been doing some analysis on the review submissions of several projects of R. However, till recently I couldn’t analyze the CRAN submission. There was cransays’ package to check package submissions which on the online documentation provided a dashboard which updated each hour. Since 2020/09/12 the status of the queues and folders of submissions are saved on a branch. Using this information and basing in script provided by Tim Taylor I’ll check how are the submissions on CRAN handled.

I’ll look at the CRAN queue, I’ll explore some time patterns and also check the meaning of those subfolder. Later I’ll go to a more practical information for people submitting a package. Lastly, we’ll see how hard is the job of the CRAN team by looking at the reliability of the Github action used.

Before all this we need preliminary work to download the data and clean it:

# Downloading the cransays repository branch history
download.file("https://github.com/lockedata/cransays/archive/history.zip", 
              destfile = "static/cransays-history.zip")
path_zip <- here::here("static", "cransays-history.zip") 
# We unzip the files to read them
dat <- unzip(path_zip, exdir = "static")
csv <- dat[grepl("*.csv$", x = dat)]
f <- lapply(csv, read.csv)
m <- function(x, y) {
  merge(x, y, sort = FALSE, all = TRUE)
}
updates <- Reduce(m, f) # Merge all files (Because the file format changed)
write.csv(updates, file = "static/cran_till_now.csv",  row.names = FALSE)
# Clean up
unlink("static/cransays-history/", recursive = TRUE)
unlink("static/cransays-history.zip", recursive = TRUE)

Once we have the data we can load it, and we load the libraries used for the analysis:

library("tidyverse")
library("lubridate")
library("hms")
path_file <- here::here("static", "cran_till_now.csv")
cran_submissions <- read.csv(path_file)
theme_set(theme_minimal()) # For plotting
col_names <- c("package", "version", "snapshot_time", "folder", "subfolder")
cran_submissions <- cran_submissions[, col_names]

The period we are going to analyze is from the beginning of the records till 2021-01-30. It includes some well earned holiday time for the CRAN team, during which submissions were not possible.

I’ve read some comments on the inconsistencies of where the holidays of the CRAN teams are reported and I couldn’t find it for previous years.

For the 4 months we are analyzing which only has one holiday period I used a screenshot found on twitter.

holidays <- data.frame(
  start = as.POSIXct("18/12/2020", format = "%d/%m/%Y", tz = "UTC"), 
  end = as.POSIXct("04/01/2021", format = "%d/%m/%Y", tz = "UTC")
)

Now that we know the holidays and have them in a single data.frame it’s time to explore and clean the data collected:

Cleaning the data

After preparing the files in one big file we can load and work with it. First steps, check the data and prepare it for what we want:

# Use appropriate class
cran_submissions$snapshot_time <- as.POSIXct(cran_submissions$snapshot_time,
                                             tz = "UTC")
# Fix subfolders structure
cran_submissions$subfolder[cran_submissions$subfolder %in% c("", "/")] <- NA
# Remove files or submissions without version number
cran_submissions <- cran_submissions[!is.na(cran_submissions$version), ]
cran_submissions <- distinct(cran_submissions, 
                             snapshot_time, folder, package, version, subfolder,
                             .keep_all = TRUE)

After loading and a preliminary cleanup, set date format, homogenize folder format, remove submissions that are not packages (yes there are pdf and other files on the queue), and remove duplicates we can start.

As always start with some checks of the data. Note: I should follow more often this advice myself, as this is the last section I’m writing on the post.

packges_multiple_versions <- cran_submissions %>% 
  group_by(package, snapshot_time) %>% 
  summarize(n = n_distinct(version)) %>% 
  filter(n != 1) %>% 
  distinct(package) %>% 
  pull(package)

There are 92 packages with multiple versions at the same time on CRAN queue.

Perhaps because packages are left in different folders (2 or even 3) at the same time:

package_multiple <- cran_submissions %>% 
  group_by(snapshot_time, package) %>% 
  count() %>% 
  group_by(snapshot_time) %>% 
  count(n) %>% 
  filter(n != 1) %>% 
  summarise(n = sum(nn)) %>% 
  ungroup()
ggplot(package_multiple) +
  geom_point(aes(snapshot_time, n), size = 1) +
  geom_rect(data = holidays, aes(xmin = start, xmax = end, ymin = 0, ymax = 6),
            alpha = 0.25, fill = "red") +
  annotate("text", x = holidays$start + (holidays$end - holidays$start)/2, 
           y = 3.5, label = "CRAN holidays") +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks", 
                   expand = expansion()) +
  scale_y_continuous(expand = expansion()) +
  labs(title = "Packages in multiple folders and subfolders", 
       x = element_blank(), y = element_blank())

This happens in 1915 snapshots of 3260, probably due to the manual labor of the CRAN reviews. I don’t really know the cause of this, it could be an error on the script recording the data, copying the data around the server. But perhaps this indicates further improvements and automatization of the process can be done.

cran_submissions <- cran_submissions %>% 
  arrange(package, snapshot_time, version, folder) %>% 
  group_by(package, snapshot_time) %>% 
  mutate(n = 1:n()) %>% 
  filter(n == n()) %>% 
  ungroup() %>% 
  select(-n)

Now we removed ~3500 records of packages with two versions remaining on the queue. Next we check packages in multiple folders but with the same version and remove them until we are left with a single one (assuming there aren’t parallel steps on the review process followed):

cran_submissions <- cran_submissions %>% 
  arrange(package, snapshot_time, folder) %>% 
  group_by(package, snapshot_time) %>% 
  mutate(n = 1:n()) %>% 
  filter(n == n()) %>% 
  ungroup() %>% 
  select(-n)

Last, we add the number of submissions, in this period, for each package:

diff0 <- structure(0, class = "difftime", units = "hours")
cran_submissions <- cran_submissions %>% 
  arrange(package, version, snapshot_time) %>% 
  group_by(package) %>% 
  # Packages last seen in queue less than 24 ago are considered same submission
  mutate(diff_time = difftime(snapshot_time,  lag(snapshot_time), units = "hour"),
         diff_time = if_else(is.na(diff_time), diff0, diff_time), # Fill NAs
         diff_v = version != lag(version),
         diff_v = ifelse(is.na(diff_v), TRUE, diff_v), # Fill NAs
         new_version = !near(diff_time, 1, tol = 24) & diff_v, 
         new_version = if_else(new_version == FALSE & diff_time == 0, 
                               TRUE, new_version),
         submission_n = cumsum(as.numeric(new_version))) %>%
  ungroup() %>% 
  select(-diff_time, -diff_v, -new_version)

As sometimes after a release there is soon a fast update to fix some bugs raised on the new features introduced, if a package isn’t seen in 24h in the queue it is considered a new submission. Also if it change the package version but not if it change the version while addressing the feedback from reviewers.

Now we have the data ready for further analysis.

CRAN load

We all know that CRAN is busy with updates to fix bugs, improve package, or with petitions to have new packages included on the repository.

A first plot we can make is showing the number of distinct packages on each moment:

cran_queue <- cran_submissions %>% 
  group_by(snapshot_time) %>% 
  summarize(n = n_distinct(package))
ggplot(cran_queue) +
  geom_rect(aes(xmin = start, xmax = end, ymin = 0, ymax = 230),
            alpha = 0.5, fill = "red", data = holidays) +
  annotate("text", x = holidays$start + (holidays$end - holidays$start)/2, 
           y = 150, label = "CRAN holidays") +
  geom_path(aes(snapshot_time, n)) +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks", 
                   expand = expansion()) +
  scale_y_continuous(expand = expansion()) +
  labs(x = element_blank(), y = element_blank(), 
       title = "Packages on CRAN review process")

We can see that there are some ups and downs, and ranges between 200 to 50.

There are some instance were the number of package on the queue have a sudden drop and then a recovery to previous levels. This as far as I know is a visual artifact.

We can also see that people do not tend to rush and push the package before the holidays. But clearly there is some build up of submissions after holidays, as the the highest number of packages on the queue is reached after holidays.

On the CRAN review process classifying package in folders seems to be part of the process:

man_colors <- RColorBrewer::brewer.pal(8, "Dark2")
names(man_colors) <- unique(cran_submissions$folder)
cran_submissions %>% 
  group_by(folder, snapshot_time) %>% 
  summarize(packages = n_distinct(package)) %>% 
  ggplot() +
  geom_rect(data = holidays, aes(xmin = start, xmax = end, ymin = 0, ymax = 200),
            alpha = 0.25, fill = "red") +
  annotate("text", x = holidays$start + (holidays$end - holidays$start)/2, 
           y = 105, label = "CRAN holidays") +
  geom_path(aes(snapshot_time, packages, col = folder)) +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks", 
                   expand = expansion()) +
  scale_y_continuous(expand = expansion()) +
  scale_color_manual(values = man_colors) +
  labs(x = element_blank(), y = element_blank(),
       title = "Packages by folder", col = "Folder") +
  theme(legend.position = c(0.6, 0.7))

The queue trend is mostly driven by newbies folder (which ranges between 25 and 150) and after holidays by the pretest folder.

Surprisingly when the queue is split by folder we don’t see those sudden drops. This might indicate that there is a clean up on some of the folders¹. What we clearly see is a clean up on holidays for all the folders, when almost all was cleared up.

Also the pretest seems to be before the newbies folder raises, so it seems like these tests are done done only to new pacakges.

There other folders do not have such an increase

after the holidays .

cran_submissions %>% 
  group_by(folder, snapshot_time) %>% 
  summarize(packages = n_distinct(package)) %>% 
  filter(snapshot_time >= holidays$start) %>% 
  ggplot() +
  geom_path(aes(snapshot_time, packages, col = folder)) +
  geom_rect(data = holidays, aes(xmin = start, xmax = end, ymin = 0, ymax = 200),
            alpha = 0.25, fill = "red") +
  annotate("text", x = holidays$start + (holidays$end - holidays$start)/2, 
           y = 105, label = "CRAN holidays") +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "1 day", 
                   expand = expansion()) +
  scale_y_continuous(expand = expansion(), limits = c(0, NA)) +
  scale_color_manual(values = man_colors) +
  labs(x = element_blank(), y = element_blank(),
       title = "Holidays", col = "Folder") +
  theme(axis.text.x = element_text(angle = 45, hjust = 1),
        legend.position = c(0.8, 0.7))

It seems like that on the 31st there was a clean up of some packages on the waiting list. And we can see the increase of submissions on the first week of January, as described previously.

Time patterns

Some people has expressed they try to submit to CRAN when there are few packages on the queue. Thus, looking when does this low moments happens could be relevant. We can look for patterns on the queue:

Note: I have few to none experience with time series, so the following plots are just using the defaults of `geom_smooth`, just omitting the holidays.

By day of the month

Looking at folder:

cran_times <- cran_submissions %>% 
  mutate(seconds = seconds(snapshot_time),
         month = month(snapshot_time),
         mday = mday(snapshot_time),
         wday = wday(snapshot_time, locale = "en_GB.UTF-8"),
         week = week(snapshot_time),
         date = as_date(snapshot_time))
cran_times %>% 
  arrange(folder, date, mday) %>% 
  filter(snapshot_time < holidays$start | snapshot_time  > holidays$end) %>% 
  group_by(folder, date, mday) %>% 
  summarize(packages = n_distinct(package),
            week = unique(week)) %>% 
  group_by(folder, mday) %>% 
  ggplot() +
  geom_smooth(aes(mday, packages, col = folder)) +
  labs(x = "Day of the month", y = "Packages", col = "Folder",
       title = "Evolution by month day") +
  scale_color_manual(values = man_colors) +
  coord_cartesian(ylim = c(0, NA), xlim = c(1, NA)) +
  scale_x_continuous(expand = expansion()) +
  scale_y_continuous(expand = expansion())

At the beginning and end of the month there is more variation on several folders (This could also be that there isn’t information of the end of December and beginning of January). There seems to be an increase of new packages submissions towards the beginning of the month and later and increase on the newbies folder by the middle of the month.

By day of the week

I first thought about this, because I was curious if there is more submission on weekends (when aficionados and open source developers might have more time) or the rest of the week.

cran_times %>% 
  filter(snapshot_time < holidays$start | snapshot_time  > holidays$end) %>% 
  group_by(folder, date, wday) %>% 
  summarize(packages = n_distinct(package),
            week = unique(week)) %>% 
  group_by(folder, wday) %>% 
  ggplot() +
  geom_smooth(aes(wday, packages, col = folder)) +
  labs(x = "Day of the week", y = "Packages", col = "Folder",
       title = "Evolution by week day") +
  scale_color_manual(values = man_colors) +
  scale_x_continuous(breaks = 1:7, expand = expansion()) +
  scale_y_continuous(expand = expansion(), limits = c(0, NA))

We see a rise towards the middle of the week of the packages on the pretest folder, indicating new packages submissions.

Other folders

There are some folders that seem to be from R Contributors. We see that some packages are on these folders:

cran_members <- c("LH", "GS", "JH")
cran_times %>% 
  filter(subfolder %in% cran_members) %>% 
  group_by(subfolder, snapshot_time) %>% 
  summarize(packages = n_distinct(package)) %>% 
  ggplot() +
  geom_smooth(aes(snapshot_time, packages, col = subfolder)) +
    labs(x = element_blank(), y = element_blank(), col = "Folder",
       title = "Packages on folders") +
  scale_y_continuous(expand = expansion(), breaks = 0:10) +
  coord_cartesian(y = c(0, NA))  +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks", 
               expand = expansion(add = 2)) +
  theme(legend.position = c(0.1, 0.8))

There doesn’t seem to be any rule about using those folders or, the work was so quick that the hourly updated data didn’t record it.

Looking for any temporal pattern on those folders isn’t worth it.

cran_times %>% 
  filter(subfolder %in% cran_members) %>% 
  group_by(subfolder, mday) %>% 
  summarize(packages = n_distinct(package)) %>% 
  ungroup() %>% 
  ggplot() +
  geom_smooth(aes(mday, packages, col = subfolder)) +
  labs(x = "Day of the month", y = "Pacakges", col = "Subfolder",
       title = "Packages on subfolers by day of the month") +
  scale_y_continuous(expand = expansion()) +
  scale_x_continuous(expand = expansion(), breaks = c(1,7,14,21,29)) +
  coord_cartesian(ylim = c(0, NA))

Low number of packages and great variability (except on those that just have 1 package on the folder) on day of month.

cran_times %>% 
  filter(subfolder %in% cran_members) %>% 
  group_by(subfolder, wday) %>% 
  summarize(packages = n_distinct(package)) %>% 
  ungroup() %>% 
  ggplot() +
  geom_smooth(aes(wday, packages, col = subfolder)) +
  labs(x = "Day of the week", y = "Pacakges", col = "Subfolder",
       title = "Evolution by week day") +
  scale_y_continuous(expand = expansion()) +
  scale_x_continuous(breaks = 1:7, expand = expansion()) +
  coord_cartesian(ylim =  c(0, NA))

There seem to be only 2 people usually working with their folders. Suppose that there aren’t a common set of rules the reviewers follow.

Information for submitters

I’ve read lots of comments recently around CRAN submission. However, with the few data available compared to open reviews on Bioconductor and rOpenSci it is hard to answer them (See those related posts). On Bioconductor and rOpenSci it is possible to see people involved, message from the reviewers and other interested parties, steps done to be accepted…

One of the big question we can provide information about with the data available is how long it will be a package on the queue:

subm <- cran_times %>%
  arrange(snapshot_time) %>% 
  select(package, version, submission_n, snapshot_time) %>% 
  group_by(package, submission_n) %>% 
  filter(row_number() %in% c(1, last(row_number()))) %>% 
  arrange(package, submission_n)

There are 429 packages that are only seen once. It might mean that it is an abandoned, delayed or rejected submissions, other might be acceptances in less than an hour².

If we look at the package submission by date we can see the quick increase of packages of packages:

rsubm <- subm %>% 
  filter(n_distinct(snapshot_time) %% 2 == 0) %>%
  select(-version) %>% 
  mutate(time = c("start", "end")) %>% 
  pivot_wider(values_from = snapshot_time, names_from = time) %>% 
  ungroup() %>% 
  mutate(r = row_number(), 
         time  =  round(difftime(end, start, units = "hour"), 0)) %>% 
  ungroup()
lv <- levels(fct_reorder(rsubm$package, rsubm$start, .fun = min, .desc = FALSE))
ggplot(rsubm) +
  geom_rect(data = holidays, aes(xmin = start, xmax = end), 
            ymin = first(lv), ymax = last(lv), alpha = 0.5, fill = "red") +
  geom_linerange(aes(y = fct_reorder(package, start, .fun = min, .desc = FALSE),
                      x = start, xmin = start, xmax = end, 
                     col = as.factor(submission_n))) + 
  labs(x = element_blank(), y = element_blank(), title = 
         "Packages on the queue", col = "Submissions") +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks", 
                   expand = expansion(add = 2)) +
  scale_colour_viridis_d() +
  theme_minimal() +
  theme(panel.grid.major.y = element_blank(),
        axis.text.y = element_blank(),
        legend.position = c(0.15, 0.7))

Some packages were submitted more than 5 times in this period. Recall the definition for submission used: a package with different version number after 24 hours or which wasn’t seen in the queue for the last 24 hours (even if they have the same version number).

Some authors do change the version number when CRAN reviewers require changes before accepting the package on CRAN while others do not and only change the version number according to their release cycle.

rsubm %>% 
  arrange(start) %>% 
  filter(start < holidays$start, # Look only before the holidays
    submission_n == 1,# Use only the first submission
    start > min(start)) %>%   # Just new submissions
  mutate(r = row_number(),
         start1 = as.numeric(seconds(start))) %>% 
  lm(start1 ~ r, data = .) %>% 
  broom::tidy() %>%  
  mutate(estimate = estimate/(60*60)) # Hours
## # A tibble: 2 x 5
##   term         estimate std.error statistic p.value
##   <chr>           <dbl>     <dbl>     <dbl>   <dbl>
## 1 (Intercept) 444325.     6321.     253047.       0
## 2 r                1.03      4.76      779.       0

More or less there is a new package submission every hour on CRAN. Despite this submission rate we can see that most submissions are on the queue a short time:

library("patchwork")
p1 <- rsubm %>% 
  group_by(package) %>% 
  summarize(time = sum(time)) %>% 
  ggplot() +
  geom_histogram(aes(time), bins = 100) +
  labs(title = "Packages total time on queue", x = "Hours", 
       y = element_blank()) +
  scale_x_continuous(expand = expansion()) +
  scale_y_continuous(expand = expansion())
p2 <- rsubm %>% 
  group_by(package) %>% 
  summarize(time = sum(time)) %>% 
  ggplot() +
  geom_histogram(aes(time), binwidth = 24) +
  coord_cartesian(xlim = c(0, 24*7)) +
  labs(subtitle = "Zoom", x = "Hours", y = element_blank()) +
  scale_x_continuous(expand = expansion(), breaks = seq(0, 24*7, by = 24)) +
  scale_y_continuous(expand = expansion()) +
  theme(panel.background = element_rect(colour = "white"))
p1 + inset_element(p2, 0.2, 0.2, 1, 1)

The accuracy of this data is not great. Because I found some packages that remained on the submission queue, and thus picked up by cransays, even after acceptance, so this might be a bit overestimated. Also, there are packages that got a speedy submission that didn’t last more than an hour, and they weren’t included.

Looking at the recorded submissions might be more accurate:

p1 <- rsubm %>% 
  group_by(package, submission_n) %>% 
  summarize(time = sum(time)) %>% 
  ggplot() +
  geom_histogram(aes(time), bins = 100) +
  labs(title = "Submission time on queue", x = "Hours", 
       y = element_blank()) +
  scale_x_continuous(expand = expansion()) +
  scale_y_continuous(expand = expansion())
p2 <- rsubm %>% 
  group_by(package, submission_n) %>% 
  summarize(time = sum(time)) %>%  
  ggplot() +
  geom_histogram(aes(time), binwidth = 24) +
  coord_cartesian(xlim = c(0, 24*7)) +
  labs(subtitle = "Zoom", x = "Hours", y = element_blank()) +
  scale_x_continuous(expand = expansion(), breaks = seq(0, 24*7, by = 24)) +
  scale_y_continuous(expand = expansion()) +
  theme(panel.background = element_rect(colour = "white"))
p1 + inset_element(p2, 0.2, 0.2, 1, 1)

Many submissions are shortly spanned. Perhaps hinting that more testing should be done before or what to expect on the review should be more clear to the authors, or that they are approved very fast, or…

There are 429 packages that are only seen once. It might mean that it is an abandoned/rejected submissions other might be acceptances in less than an hour.

If we look at the folders of each submission well see different picture:

subm2 <- cran_times %>%
  group_by(package, submission_n, folder) %>% 
  arrange(snapshot_time) %>% 
  select(package, version, submission_n, snapshot_time, folder) %>% 
  filter(row_number() %in% c(1, last(row_number()))) %>% 
  arrange(submission_n)
rsubm2 <- subm2 %>% 
  filter(n_distinct(snapshot_time) %% 2 == 0) %>%
  mutate(time = c("start", "end")) %>% 
  pivot_wider(values_from = snapshot_time, names_from = time) %>% 
  ungroup() %>% 
  mutate(r = row_number(), 
         time  =  round(difftime(end, start, units = "hour"), 0)) %>% 
  ungroup() %>% 
  filter(!is.na(start), !is.na(end))
lv <- levels(fct_reorder(rsubm2$package, rsubm2$start, .fun = min, .desc = FALSE))
ggplot(rsubm2) +
  geom_rect(data = holidays, aes(xmin = start, xmax = end), 
            ymin = first(lv), ymax = last(lv), alpha = 0.5, fill = "red") +
  geom_linerange(aes(y = fct_reorder(package, start, .fun = min, .desc = FALSE),
                      x = start, xmin = start, xmax = end, col = folder)) + 
  labs(x = element_blank(), y = element_blank(), title = 
         "Packages on the queue") +
  scale_color_manual(values = man_colors) +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks", 
               expand = expansion(add = 2)) +
  labs(col = "Folder") +
  theme_minimal() +
  theme(panel.grid.major.y = element_blank(),
        axis.text.y = element_blank(),
        legend.position = c(0.2, 0.7))

It looks like some packages take a long time to change folder, perhaps the maintainers have troubles fixing the issues pointed by the reviewers, or don’t have time to deal with them. Some packages are recorded in just 1 folder and some other go through multiple folders:

rsubm2 %>% 
  group_by(package, submission_n) %>% 
  summarize(n_folder = n_distinct(folder)) %>% 
  ggplot() + 
  geom_histogram(aes(n_folder), bins = 5) +
  labs(title = "Folders by submission", x = element_blank(), 
       y = element_blank())

Most submissions end up in one folder, but some go up to 5 folders.

Let’s see the most 5 common folders process of submissions:

compact_folders <- function(x) {
  y <- x != lag(x)
  y[1] <- TRUE
  x[y]
}
cran_times %>% 
  group_by(package, submission_n) %>% 
  summarize (folder = list(compact_folders(folder))) %>% 
  ungroup() %>% 
  count(folder, sort = TRUE) %>% 
  top_n(5) %>% 
  rename(Folders = folder, Frequency = n) %>% 
  as.data.frame()
##            Folders Frequency
## 1          pretest      1433
## 2 pretest, inspect       422
## 3          inspect       301
## 4 pretest, newbies       279
## 5          newbies       245

As expected the pretest and newbies are one of the most frequent folders.

Another way of seeing whether it is a right moment to submit your package, aside of how many packages are on the queue, is looking how much activity there is:

subm3 <- cran_times %>%
  arrange(snapshot_time) %>% 
  group_by(package) %>% 
  mutate(autor_change = submission_n != lag(submission_n),
         cran_change = folder != lag(folder)) %>% 
  mutate(autor_change = ifelse(is.na(autor_change), TRUE, autor_change),
         cran_change = ifelse(is.na(cran_change), FALSE, cran_change)) %>% 
  mutate(cran_change = case_when(subfolder != lag(subfolder) ~ TRUE,
                                 TRUE ~ cran_change)) %>% 
  ungroup()
subm3 %>% 
  group_by(snapshot_time) %>% 
  summarize(autor_change = sum(autor_change), cran_change = sum(cran_change)) %>% 
  filter(row_number() != 1) %>% 
  filter(autor_change != 0 | cran_change != 0) %>% 
  ggplot() +
  geom_rect(data = holidays, aes(xmin = start, xmax = end), 
            ymin = -26, ymax = 26, alpha = 0.5, fill = "grey") +
  geom_point(aes(snapshot_time, autor_change), fill = "blue", size = 0) +
  geom_area(aes(snapshot_time, autor_change), fill = "blue") +
  geom_point(aes(snapshot_time, -cran_change), fill = "red", size = 0) +
  geom_area(aes(snapshot_time, -cran_change), fill = "red") +
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks", 
                   expand = expansion(add = 2)) +
  scale_y_continuous(expand = expansion(add = c(0, 0))) + 
  coord_cartesian(ylim = c(-26, 26)) +
  annotate("text", label = "CRAN's", y = 20, x = as_datetime("2020/11/02")) +
  annotate("text", label = "Maintainers'", y = -20, x = as_datetime("2020/11/02")) +
  labs(y = "Changes", x = element_blank(), title = "Activity on CRAN:")

On this plot we can see that changes on folders or submissions are not simultaneous. But they are quite frequent.

Review process

There is a scheme about how does the review process work. However, it has been pointed out that it needs an update.

We’ve seen which folders go before which ones, but we haven’t looked up what is the last folder in which package appear:

cran_times %>% 
  ungroup() %>% 
  group_by(package, submission_n) %>% 
  arrange(snapshot_time) %>% 
  filter(1:n() == last(n())) %>% 
  ungroup() %>% 
  count(folder, sort = TRUE) %>% 
  knitr::kable(col.names = c("Last folder", "Submissions"))

Last folder	Submissions
pretest	1653
newbies	981
inspect	890
recheck	555
publish	469
waiting	441
human	332
pending	225

We can see that many submissions were left at the pretest, and just a minority on the human or publish folders.

Time it takes to disappear from the system

One of the motivations to do this post was a question on R-pkg-devel, about how long does it usually take for a package to be accepted on CRAN. We can look how long does each submission take until it is removed from the queue:

package_submissions <- cran_times %>% 
  group_by(package, submission_n) %>% 
  summarise(submission_period = difftime(max(snapshot_time), 
                                         min(snapshot_time), 
                                         units = "hour"),
            submission_time = min(snapshot_time)) %>% 
  ungroup() %>% 
  filter(submission_period != 0)

This is a good approximation about how long does it take a package to be accepted or rejected, but some packages remain on the queue after they are accepted and appear on CRAN. Joining this data with data from metacran we could know how often does this happend. But I leave that for the reader or some other posts. Let’s go back to time spend on the queue:

package_submissions %>% 
  # filter(submission_time < holidays$start) %>% 
  ggplot() +
  geom_point(aes(submission_time, submission_period, col = submission_n)) +
  geom_rect(data = holidays, aes(xmin = start, xmax = end),
            ymin = 0, ymax = 3500, alpha = 0.5, fill = "red") + 
  scale_x_datetime(date_labels = "%Y/%m/%d", date_breaks = "2 weeks",
                   expand = expansion(add = 10)) +
  scale_y_continuous(expand = expansion(add = 10)) +
  labs(title = "Time on the queue according to the submission",
       x = "Submission", y = "Time (hours)", col = "Submission") +
  theme(legend.position = c(0.5, 0.8))

These diagonals suggest that the work is done in batches in a day or an afternoon. And the prominent diagonal after holidays are packages still on the queue.

If we summarize by day and take the median of all the first package submission we can see how much long is a package on the queue:

package_submissions %>% 
  filter(submission_n == 1) %>% 
  ungroup() %>%
  mutate(d = as.Date(submission_time)) %>%
  group_by(d) %>% 
  summarize(m = median(submission_period)) %>% 
  ggplot() +
  geom_rect(data = holidays, aes(xmin = as.Date(start), xmax = as.Date(end)),
            ymin = 0, ymax = 80, alpha = 0.5, fill = "red") + 
  geom_smooth(aes(d, m)) +
  coord_cartesian(ylim = c(0, NA)) +
  scale_x_date(date_labels = "%Y/%m/%d", date_breaks = "2 weeks",
                   expand = expansion(add = 1)) +
  labs(x = element_blank(), y = "Daily median time in queue (hours)", 
       title = "Submission time")

We can see that usually it takes more than a day to disappear from the queue for a new package submitted on CRAN.

There is a lot of variation among submissions:

package_submissions %>% 
  group_by(submission_n) %>% 
  mutate(submission_n = as.character(submission_n)) %>% 
  ggplot() +
  geom_jitter(aes(submission_n, submission_period), height = 0) +
  scale_y_continuous(limits = c(1, NA), expand = expansion(add = c(1, 10)),
                     breaks = seq(0,  4550, by = 24*7)) +
  labs(title = "Submission time in queue", y = "Hours", x = element_blank())
## Warning: Removed 142 rows containing missing values (geom_point).

Surprisingly sometimes a submissions goes missing from the folders for some days (I checked with one package I submitted and it doesn’t appear for 7 days, although it was on the queue). This might affect this analysis as it will count them as new submissions but some of them won’t be.

package_submissions %>% 
  filter(submission_period != 0) %>% 
  group_by(submission_n) %>% 
  mutate(submission_n = as.character(submission_n)) %>% 
  filter(n() > 5) %>% 
  summarize(median = round(median(submission_period), 2)) %>% 
  knitr::kable(col.names = c("Submission", "Median time (h)"))

Submission	Median time (h)
1	36.13 hours
2	18.27 hours
3	16.47 hours
4	11.27 hours
5	13.37 hours
6	38.08 hours

So it usually takes more than a day for new packages and later is around 12 hours.

To set in context the work done by CRAN checking system, which is the one that helps to keep the high quality of the packages, let’s explore other checking system: GitHub actions.

GitHub action reliability

The data of this post was collected using Github actions by cransays. Well use this data to test how reliable GitHub actions are.

gha <- cbind(cran_times[, c("month", "mday", "wday", "week")], 
      minute = minute(cran_submissions$snapshot_time), 
      hour = hour(cran_times$snapshot_time),
      type = "cransays") %>% 
  distinct()
gha %>% 
  ggplot() +
  geom_violin(aes(as.factor(hour), minute)) +
  scale_y_continuous(expand = expansion(add = 0.5), 
                     breaks = c(0, 15, 30, 45, 60), limits = c(0, 60)) +
  scale_x_discrete(expand = expansion())  +
  labs(x = "Hour", y = "Minute", title = "Daily variation")

There seems to be a lower limit around 10 minutes except for some builds that I think were manually triggered. Aside from this, there is usually low variation and the process end around ~ 15 minutes but it can end much later. This is just for one simple script scrapping a site. Compared to thousands of packages builds and checks it is much simpler.

And last how reliable it is?

We can compare how many hours are between the first and the last report and how many do we have recorded. If we have less this indicate errors on GHA.

So the script and github actions worked on ~96.9646576% of the times.

These numbers are great, but on CRAN and Bioconductor all packages are checked daily on several OS consistently.

Conclusions

Some of the most important points from this post:

Some packages appear on several folders and even there are times that multiple versions of a package are on the queue.
Most submission happens in the first days of the week and towards the beginning of the month.
Most of the submissions disappear from the CRAN queue in less than a day but new submission take around 36 hours.
There’s a new package submission to CRAN every hour.
In later submissions time in the queue is considerably shorter.
It was impossible to know when there was a reply from CRAN, as no information is provided.
Not possible to know when a package has all OK before it hits CRAN as some packages remain in the queue even after acceptance.

Last, I compare the review system with other reviews of R software like Bioconductor and rOpenSci.

One big difference between CRAN and Bioconductor or rOpenSci is that even if your package is already included each time you want to fix something it gets reviewed by someone. This ensures a high quality of the packages, as well as increases the work for the reviewers.

Also, the list of reviewers as far as I know is just 5 people who also are part of the developers maintaining and developing R. In Bioconductor it is the same (except they do not to take care of R itself) but in rOpenSci this doesn’t happen that way.

The next big difference is the lack of transparency on the process of the review itself. Perhaps because CRAN started earlier (1997) while Bioconductor in 2002 and rOpenSci much later. On CRAN with the information available we don’t know the steps to be accepted beyond the pretest and the manual review. We don’t know when the package is accepted or rejected, what is the content of the feedback to the maintainer (or when there is feedback and how long does the maintainer get to address those changes). It is no clear how does the process work. Additionally, the reviewer’s work seems highly manual as we found some duplicated packages on the queue.

Further automatization and transparency on the process could help reduce the load on the reviewers, as well as increasing the number of reviewers. A public review could help reduce the burden to CRAN reviewers as outsiders could help solving errors (although this is somewhat already fulfilled by the mailing list R-package-devel), and would help notice and find a compromise on inconsistencies between reviews. As anecdotal evidence I submitted two packages one shortly after the first, on the second package they asked me to change some URLS that on the first I wasn’t required to change.

Another difference between these three repositories are the manuals. It seems that CRAN repository is equated to be R, so the manual for writing R extensions is under cran.r-project.org, while this is about extending R and can and does happen outside CRAN.

The CRAN policies changes without notice to the existing developers. Sending an email to the maintainers or R-announce mailing list would help developers to notice policy changes. Developers had to create a policy watch and other resources to keep up with CRAN changes that not only affect them on submitting a package but also on packages already included on CRAN.

The CRAN reviewers are involved on multiple demanding tasks: their own regular jobs and outside work (familiar, friend, other interests) commitments, and then, R development and maintenance, CRAN reviews and maintenance, R-journal³. One possible solution to reduce the burden for them is increase the number of reviewers. Perhaps a mentorship program to review packages or a guideline to what to check training and would help reduce the pressure on the current volunteers.

The peace and work of the maintainers as seen on this analysis is huge, and much more that cannot be seen with this data. Many thanks to all the volunteers that maintain it, those who donate to the R Foundation and the employers of those volunteers that make possible CRAN and R.

Reproducibility

## NULL
## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.3 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2021-08-25                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package      * version     date       lib source                              
##  assertthat     0.2.1       2019-03-21 [1] CRAN (R 4.0.1)                      
##  backports      1.2.1       2020-12-09 [1] CRAN (R 4.0.1)                      
##  blogdown       1.3         2021-04-14 [1] CRAN (R 4.0.1)                      
##  bookdown       0.22        2021-04-22 [1] CRAN (R 4.0.1)                      
##  broom          0.7.6       2021-04-05 [1] CRAN (R 4.0.1)                      
##  bslib          0.2.5       2021-05-12 [1] CRAN (R 4.0.1)                      
##  cellranger     1.1.0       2016-07-27 [1] CRAN (R 4.0.1)                      
##  cli            2.5.0       2021-04-26 [1] CRAN (R 4.0.1)                      
##  colorspace     2.0-1       2021-05-04 [1] CRAN (R 4.0.1)                      
##  crayon         1.4.1       2021-02-08 [1] CRAN (R 4.0.1)                      
##  DBI            1.1.1       2021-01-15 [1] CRAN (R 4.0.1)                      
##  dbplyr         2.1.1       2021-04-06 [1] CRAN (R 4.0.1)                      
##  digest         0.6.27      2020-10-24 [1] CRAN (R 4.0.1)                      
##  dplyr        * 1.0.6       2021-05-05 [1] CRAN (R 4.0.1)                      
##  ellipsis       0.3.2       2021-04-29 [1] CRAN (R 4.0.1)                      
##  evaluate       0.14        2019-05-28 [1] CRAN (R 4.0.1)                      
##  fansi          0.5.0       2021-05-25 [1] CRAN (R 4.0.1)                      
##  farver         2.1.0       2021-02-28 [1] CRAN (R 4.0.1)                      
##  forcats      * 0.5.1       2021-01-27 [1] CRAN (R 4.0.1)                      
##  fs             1.5.0       2020-07-31 [1] CRAN (R 4.0.1)                      
##  generics       0.1.0       2020-10-31 [1] CRAN (R 4.0.1)                      
##  ggplot2      * 3.3.5       2021-06-25 [1] CRAN (R 4.0.1)                      
##  glue           1.4.2       2020-08-27 [1] CRAN (R 4.0.1)                      
##  gtable         0.3.0       2019-03-25 [1] CRAN (R 4.0.1)                      
##  haven          2.4.1       2021-04-23 [1] CRAN (R 4.0.1)                      
##  here           1.0.1       2020-12-13 [1] CRAN (R 4.0.1)                      
##  highr          0.9         2021-04-16 [1] CRAN (R 4.0.1)                      
##  hms          * 1.0.0       2021-01-13 [1] CRAN (R 4.0.1)                      
##  htmltools      0.5.1.1     2021-01-22 [1] CRAN (R 4.0.1)                      
##  httr           1.4.2       2020-07-20 [1] CRAN (R 4.0.1)                      
##  jquerylib      0.1.4       2021-04-26 [1] CRAN (R 4.0.1)                      
##  jsonlite       1.7.2       2020-12-09 [1] CRAN (R 4.0.1)                      
##  knitr          1.33        2021-04-24 [1] CRAN (R 4.0.1)                      
##  labeling       0.4.2       2020-10-20 [1] CRAN (R 4.0.1)                      
##  lattice        0.20-41     2020-04-02 [1] CRAN (R 4.0.1)                      
##  lifecycle      1.0.0       2021-02-15 [1] CRAN (R 4.0.1)                      
##  lubridate    * 1.7.10.9000 2021-06-12 [1] Github (tidyverse/lubridate@1e0d66f)
##  magrittr       2.0.1       2020-11-17 [1] CRAN (R 4.0.1)                      
##  Matrix         1.3-2       2021-01-06 [1] CRAN (R 4.0.1)                      
##  mgcv           1.8-35      2021-04-18 [1] CRAN (R 4.0.1)                      
##  modelr         0.1.8       2020-05-19 [1] CRAN (R 4.0.1)                      
##  munsell        0.5.0       2018-06-12 [1] CRAN (R 4.0.1)                      
##  nlme           3.1-152     2021-02-04 [1] CRAN (R 4.0.1)                      
##  patchwork    * 1.1.1       2020-12-17 [1] CRAN (R 4.0.1)                      
##  pillar         1.6.1       2021-05-16 [1] CRAN (R 4.0.1)                      
##  pkgconfig      2.0.3       2019-09-22 [1] CRAN (R 4.0.1)                      
##  purrr        * 0.3.4       2020-04-17 [1] CRAN (R 4.0.1)                      
##  R6             2.5.0       2020-10-28 [1] CRAN (R 4.0.1)                      
##  RColorBrewer   1.1-2       2014-12-07 [1] CRAN (R 4.0.1)                      
##  Rcpp           1.0.6       2021-01-15 [1] CRAN (R 4.0.1)                      
##  readr        * 1.4.0       2020-10-05 [1] CRAN (R 4.0.1)                      
##  readxl         1.3.1       2019-03-13 [1] CRAN (R 4.0.1)                      
##  reprex         2.0.0       2021-04-02 [1] CRAN (R 4.0.1)                      
##  rlang          0.4.11      2021-04-30 [1] CRAN (R 4.0.1)                      
##  rmarkdown      2.9         2021-06-15 [1] CRAN (R 4.0.1)                      
##  rprojroot      2.0.2       2020-11-15 [1] CRAN (R 4.0.1)                      
##  rstudioapi     0.13        2020-11-12 [1] CRAN (R 4.0.1)                      
##  rvest          1.0.0       2021-03-09 [1] CRAN (R 4.0.1)                      
##  sass           0.4.0       2021-05-12 [1] CRAN (R 4.0.1)                      
##  scales         1.1.1       2020-05-11 [1] CRAN (R 4.0.1)                      
##  sessioninfo    1.1.1       2018-11-05 [1] CRAN (R 4.0.1)                      
##  stringi        1.6.2       2021-05-17 [1] CRAN (R 4.0.1)                      
##  stringr      * 1.4.0       2019-02-10 [1] CRAN (R 4.0.1)                      
##  tibble       * 3.1.2       2021-05-16 [1] CRAN (R 4.0.1)                      
##  tidyr        * 1.1.3       2021-03-03 [1] CRAN (R 4.0.1)                      
##  tidyselect     1.1.1       2021-04-30 [1] CRAN (R 4.0.1)                      
##  tidyverse    * 1.3.1       2021-04-15 [1] CRAN (R 4.0.1)                      
##  utf8           1.2.1       2021-03-12 [1] CRAN (R 4.0.1)                      
##  vctrs          0.3.8       2021-04-29 [1] CRAN (R 4.0.1)                      
##  viridisLite    0.4.0       2021-04-13 [1] CRAN (R 4.0.1)                      
##  withr          2.4.2       2021-04-18 [1] CRAN (R 4.0.1)                      
##  xfun           0.24        2021-06-15 [1] CRAN (R 4.0.1)                      
##  xml2           1.3.2       2020-04-23 [1] CRAN (R 4.0.1)                      
##  yaml           2.2.1       2020-02-01 [1] CRAN (R 4.0.1)                      
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library

Or a problem with ggplot2 representing a sudden value that is much different from those around them.↩︎
Which now I cannot find the evidence to link to. If anyone finds the tweet I would appreciate it.↩︎
I’m not aware of anyone whose full job is just R reviewing.↩︎

rOpenSci submissions

Wed, 02 Sep 2020 00:00:00 +0000

Following last post on Bioconductor I wanted to analyze another venue where code reviews are made: rOpenSci. There are some other analysis of the the reviews made by rOpenSci themselves:

The first post in the series will explain how I rectangled onboarding. The second post will give some clues as to how to quantify the work represented by rOpenSci onboarding. The third and last post will use tidy text analysis of onboarding threads to characterize the social weather of onboarding.

rOpenSci review process has some of the differences:

One can ask before hand if the package submitted fits the scope of rOpenSci. In addition the review is done by two volunteers usually not affiliated with rOpenSci but with an editor that has the final decision Also there is an encouragement on a dialog and iteration to improve the package. Lastly, the build process is perform by third parties (Github actions, travis, appveyor …) and the results are not reported on the issue.

Despite all these differences following the same methods I downloaded the data on 2020/09/02 to analyze similarly the submissions.

We can see that there are fewer issues opened on rOpenSci but also the timeline is shorter. One difference is that there seems to be more events after the initial activity on the issues.

If we look at the editors involved on the issues we can see who has commented more and in more issues

It seems like there are 10 comments per each issue an editor is involved with. Of course, this will change for people submitting software:

Here I kept the same threshold to show the names of users. There are more names, indicating more people commenting and involved on the issues. I’m surprised to appear on this graph because I haven’t been much involved with rOpenSci. Stefanie Butland as the community manager of rOpenSci comments more and is involved in more issues than regular users.

Now that we know who is involved comenting, what about the other events?

First thing, more people are mentioned between them than on Bioconductor. Also there seems to be a lot more of cross-references than on Bioconductor.

The issues do have a highly social component with lots of mentions, subscriptions and comments. It is generally rare to unsubscribe from an issue or to have the issue added to a project.

Here we don’t see a clear pattern on the events, but most issues have few events (median events of 61).

It is fairly similar to Bioconductor. We can see that issues have fewer events than on Bioconductor. This is because there isn’t any bot replying with each update to the repository neither a report of the build with every version increment.

Most issues have all the events on the first weeks.

Editor

As mentioned the editor is assigned to find the reviewers and have the final decision on the package.

There seem to be few assigned users, and in some cases there are like two users assigned. This might be because the reviewers are also assigned that issue or something else.

Editors are fairly more involved on the issues commenting around 11 times.

We can see that one presubmission issue took longer to decide than most of the submissions! In general submissions show the same behavior as in Bioconductor: Most of them have lots of events on a relatively short period of time. Some submissions take longer to close and remain without events for long periods of time.

Here are some differences on the number of different events per issue and different users involved. There are more users and events involved with issues on rOpenSci than on Bioconductor. This is partially expected as the review are done by more people but I expected a similar pattern of events per issue.

We can see that users and events are more distributed.

As in Bioconductor the more people involved more events per issue.

Who does each action ?

We can look now at who performs what, we know there are 522 participants:

The top 35 people involved is dominated by editors. Surprisingly editors do also cross-reference other issues.

If we look at how many comments there are on each issue per author and editor we can get a sense of how much work it takes for editors:

Here we can see that there are more comments from authors and editors.

Some issues have fairly large number of comments after being closed. Perhaps discussing other alternatives or comparing to other existing software.

Mentions

There are many more mentions than on Bioconductor. So what do these people do when added to the conversation ?

People mentioned bring their experience by commenting, and cross-referencing some other issues.

Labels

rOpenSci uses labels much more than Bioconductor, with a total of 65 different labels. It also uses them to mark on which step of the review process each issue is. We can see that it got expanded with time, from initially just three labels to the current 6-7 labels:

But labels are also used to indicate to which is the topic of each package:

Surprisingly the focus seems to be on data-access followed by geospatial and data-extraction:

Topic	n
data-access	83
geospatial	30
data-extraction	26
reproducibility	25
data-munging	17
text-mining	17
data-retrieval	13
earth-science	13
climate-data	11
literature	11

Labels are also used to classify topics (and initially if it had an editor assigned)

Other labels assigned, seem to be for packages or some other related to the process. Surprisingly there have been few submissions to MEE and JOSS according to the “pub:” labels.

Looking at the labels for each step can compute the median time required to get them:

Step	Median days	Total days
1/editor-checks	2.1	2.1
2/seeking-reviewer(s)	2.5	4.6
3/reviewer(s)-assigned	10.8	15.4
4/review(s)-in-awaiting-changes	26.0	41.4
5/awaiting-reviewer(s)-response	30.5	71.9
6/approved	17.0	88.9

We see that the slowest step is awaiting reviewers response, right after awaiting changes from the authors. According to this the review process is long and it takes around 3 months to get a package approved. However, if we look for each issue how much time does it take to get the next label we see another picture:

name	Median days	Total days
1/editor-checks	2.1	2.1
2/seeking-reviewer(s)	1.9	4.0
3/reviewer(s)-assigned	6.4	10.4
4/review(s)-in-awaiting-changes	25.0	35.4
5/awaiting-reviewer(s)-response	17.1	52.5
6/approved	12.2	64.7

Now the longest step is changing the packages with the feedback from the reviewers, followed by awaiting the reviewers comments but that is almost half of the time it was. The other times do not change much. The total time is still around 2 months, which is the double of what it takes to get a package accepted in Bioconductor.

Conclusions

Following the steps/labels the review process is similar enough to be able to compare the review process of Bioconductor and rOpenSci. Not having information about the build and check status of packages makes harder to compare some steps and the stage of the package upon submission. On some early submissions it was editor’s duties to review the packages like in Bioconductor. It was later abandoned in favor of two external reviewers. The reviews on the rOpenSci are handled by more people which makes submission to take longer (probably also because usually there are two reviewers) and because there might be a clarification of changes and a dialog after the first review.

In general it takes longer to get the packages approved than on Bioconductor.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.1 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2020-10-26                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package      * version    date       lib source                             
##  assertthat     0.2.1      2019-03-21 [1] CRAN (R 4.0.1)                     
##  backports      1.1.10     2020-09-15 [1] CRAN (R 4.0.1)                     
##  blob           1.2.1      2020-01-20 [1] CRAN (R 4.0.1)                     
##  blogdown       0.21       2020-10-11 [1] CRAN (R 4.0.1)                     
##  bookdown       0.21       2020-10-13 [1] CRAN (R 4.0.1)                     
##  broom          0.7.2      2020-10-20 [1] CRAN (R 4.0.1)                     
##  cellranger     1.1.0      2016-07-27 [1] CRAN (R 4.0.1)                     
##  cli            2.1.0      2020-10-12 [1] CRAN (R 4.0.1)                     
##  colorspace     1.4-1      2019-03-18 [1] CRAN (R 4.0.1)                     
##  crayon         1.3.4      2017-09-16 [1] CRAN (R 4.0.1)                     
##  DBI            1.1.0      2019-12-15 [1] CRAN (R 4.0.1)                     
##  dbplyr         1.4.4      2020-05-27 [1] CRAN (R 4.0.1)                     
##  digest         0.6.26     2020-10-17 [1] CRAN (R 4.0.1)                     
##  dplyr        * 1.0.2      2020-08-18 [1] CRAN (R 4.0.1)                     
##  ellipsis       0.3.1      2020-05-15 [1] CRAN (R 4.0.1)                     
##  evaluate       0.14       2019-05-28 [1] CRAN (R 4.0.1)                     
##  fansi          0.4.1      2020-01-08 [1] CRAN (R 4.0.1)                     
##  farver         2.0.3      2020-01-16 [1] CRAN (R 4.0.1)                     
##  forcats      * 0.5.0      2020-03-01 [1] CRAN (R 4.0.1)                     
##  fs             1.5.0      2020-07-31 [1] CRAN (R 4.0.1)                     
##  generics       0.0.2      2018-11-29 [1] CRAN (R 4.0.1)                     
##  ggplot2      * 3.3.2      2020-06-19 [1] CRAN (R 4.0.1)                     
##  ggrepel      * 0.8.2      2020-03-08 [1] CRAN (R 4.0.1)                     
##  glue           1.4.2      2020-08-27 [1] CRAN (R 4.0.1)                     
##  gtable         0.3.0      2019-03-25 [1] CRAN (R 4.0.1)                     
##  haven          2.3.1      2020-06-01 [1] CRAN (R 4.0.1)                     
##  here           0.1        2017-05-28 [1] CRAN (R 4.0.1)                     
##  highr          0.8        2019-03-20 [1] CRAN (R 4.0.1)                     
##  hms            0.5.3      2020-01-08 [1] CRAN (R 4.0.1)                     
##  htmltools      0.5.0      2020-06-16 [1] CRAN (R 4.0.1)                     
##  httr           1.4.2      2020-07-20 [1] CRAN (R 4.0.1)                     
##  jsonlite       1.7.1      2020-09-07 [1] CRAN (R 4.0.1)                     
##  knitr          1.30       2020-09-22 [1] CRAN (R 4.0.1)                     
##  labeling       0.4.2      2020-10-20 [1] CRAN (R 4.0.1)                     
##  lifecycle      0.2.0      2020-03-06 [1] CRAN (R 4.0.1)                     
##  lubridate      1.7.9      2020-06-08 [1] CRAN (R 4.0.1)                     
##  magrittr       1.5.0.9000 2020-08-21 [1] Github (tidyverse/magrittr@1d0559d)
##  modelr         0.1.8      2020-05-19 [1] CRAN (R 4.0.1)                     
##  munsell        0.5.0      2018-06-12 [1] CRAN (R 4.0.1)                     
##  patchwork    * 1.0.1      2020-06-22 [1] CRAN (R 4.0.1)                     
##  pillar         1.4.6      2020-07-10 [1] CRAN (R 4.0.1)                     
##  pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.0.1)                     
##  purrr        * 0.3.4      2020-04-17 [1] CRAN (R 4.0.1)                     
##  R6             2.4.1      2019-11-12 [1] CRAN (R 4.0.1)                     
##  RColorBrewer   1.1-2      2014-12-07 [1] CRAN (R 4.0.1)                     
##  Rcpp           1.0.5      2020-07-06 [1] CRAN (R 4.0.1)                     
##  readr        * 1.4.0      2020-10-05 [1] CRAN (R 4.0.1)                     
##  readxl         1.3.1      2019-03-13 [1] CRAN (R 4.0.1)                     
##  reprex         0.3.0      2019-05-16 [1] CRAN (R 4.0.1)                     
##  rlang          0.4.8      2020-10-08 [1] CRAN (R 4.0.1)                     
##  rmarkdown      2.5        2020-10-21 [1] CRAN (R 4.0.1)                     
##  rprojroot      1.3-2      2018-01-03 [1] CRAN (R 4.0.1)                     
##  rstudioapi     0.11       2020-02-07 [1] CRAN (R 4.0.1)                     
##  rvest          0.3.6      2020-07-25 [1] CRAN (R 4.0.1)                     
##  scales         1.1.1      2020-05-11 [1] CRAN (R 4.0.1)                     
##  sessioninfo    1.1.1      2018-11-05 [1] CRAN (R 4.0.1)                     
##  stringi        1.5.3      2020-09-09 [1] CRAN (R 4.0.1)                     
##  stringr      * 1.4.0      2019-02-10 [1] CRAN (R 4.0.1)                     
##  tibble       * 3.0.4      2020-10-12 [1] CRAN (R 4.0.1)                     
##  tidyr        * 1.1.2      2020-08-27 [1] CRAN (R 4.0.1)                     
##  tidyselect     1.1.0      2020-05-11 [1] CRAN (R 4.0.1)                     
##  tidyverse    * 1.3.0      2019-11-21 [1] CRAN (R 4.0.1)                     
##  vctrs          0.3.4      2020-08-29 [1] CRAN (R 4.0.1)                     
##  viridisLite    0.3.0      2018-02-01 [1] CRAN (R 4.0.1)                     
##  withr          2.3.0      2020-09-22 [1] CRAN (R 4.0.1)                     
##  xfun           0.18       2020-09-29 [1] CRAN (R 4.0.1)                     
##  xml2           1.3.2      2020-04-23 [1] CRAN (R 4.0.1)                     
##  yaml           2.2.1      2020-02-01 [1] CRAN (R 4.0.1)                     
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library

Bioconductor submissions: reviews

Fri, 31 Jul 2020 00:00:00 +0000

First post is on Bioconductor submissions raised some questions comments but at the time I didn’t have a good way to answer them:

issues get closed after they got assigned a reviewer and before the reviewer actually gets a chance to start the review.
issues assigned to multiple people or issues that switched reviewers

To answer both of them I needed more information. On the previous post I only gathered the information of the state of the issues at that moment. This excluded label changes, reviewer changes, renaming the issues, who commented on the issues and many more things. To retrieve these information from github I developed a new package {socialGH} which downloads it from Github to make this analysis possible.

I gathered all the information on issues using {socialGH} on the 2020/08/18 and stored them in a tidy format. Now that we have more data about the issues we can plot them similarly to what we did on the previous post:

We can see that sometimes the issues remained silent for several months and then had a high level of events, or a single one (closing mainly).

Compare to the previous version it is surprising to see that one of the earliest issues still gets new events to date. Apparently issue 51, and others, are being used to test the Bioconductor builder or the bot used to automate the process.

We can see that some reviewers comment more and are involved in more issues, but there is a group of fairly similar comments and issues.

Clearly this will not be the same with users who submit packages:

We can clearly see that most users are mainly only involved on submitting packages on one issue, but some people participate a lot and/or in several issues.

Events

We cab count how many users and issues have been involved with each event type:

We can see that few issues are locked (remember that it is different than closing the issue) or comments deleted. There is an almost equal amount of labeling and unlabeling, performed by few people but on lots of issues. There is a group of mentioning, commenting and subscribing too. Many people is involved on commenting and creating submissions.

Looking a bit further on the issues we can look at the events that take place:

We can see that most issues have few events which agrees with the previous findings that issues are handled fairly and expeditiously.

The most common event are comments, mentions and the rarest events are deleting comments, locking or referencing them.

We can see that most issues get around 50 events, but we can clearly see lots of issues that receive very few events.

However, looking at the distribution along time most of the events are on the first 6 months after opening the issue.

Some submissions have many events while others have few events but last longer. This is without looking at issues that were closed and later reopened, or the involvement of bioc-issue-bot. We can see that in a short amount of days a lot of events can be triggered. This is mostly that for each version update there are at least two messages on the issue.

Reviews

Most of the differences are due to the automatic or manual review of the submissions.

After passing some preliminary checks a reviewer is assigned to manually review the package:

We can see that usually some adjustment to reviewers is made usually changing them but some times they are added to have more than 1 reviewer (194). Almost all the submitted packages without a reviewer (537 submissions) were rejected except for three: 81, 82, 83 that were reviewed by vobencha even if not officially assigned. Some of these were rejected because they didn’t pass some automatic check and other after preliminary inspection.

If some of these preliminary checks aren’t corrected the issue is closed:

We can see that is common to close and reopen issues, but they are generally done by the same people (the reviewer, the original submitter or the bot).

On submissions with a reviewer assigned that were closed by someone (not the submitter), we can see here that reviews with more than one reviewer do not lead to significantly more comments from the reviewers.

Although it seems like this leads to have the package approved.

We can see that there are slightly more comments from reviewers on the approved packages. Perhaps because they provide more feedback once the review starts.

We can see that the percentage of approved on those issues where the reviewer commented is fairly similar and around 80% or above. However there is also at least a 5% of submissions that are closed despite the comments from reviewers. Usually this has to be with unresponsiveness from the submitter:

We can see that on the approved packages there are usually more events on the same time period. So make sure to follow the advice of the reviewers and the bot and make all the errors and warnings disappear.

While most issues have at least 8 different events they usually have 4 users involved. Presumably the creator of the issue, bioc-issue-bot, a reviewer and someone else.

The more users involved, more different type of events are triggered. Presumably more people get subscribed or is mentioned.

As expected the more users are involved in an issue more events are produced. We can also see the submissions that are closed by the bioc-issue-bot with few comments and users.

Who does each action ?

We can look now at who performs what, we know there are 951 participants:

Here I cut the top 35 people who have triggered more than events. We can clearly see who are official reviewers (as seen on the previous post) because they labeled and unlabeled issues. bioc-issue-bot is an special user that makes lots of comments and assigning reviewers to issues. We can also see that it previously renamed the issues or unassigned reviewers.

Comments by users (excluding the bot), usually only the author, the reviewer comment, frequently mtmorgan also suggests.

We can see that Martin Morgan has commented on almost all the submissions. The odd issue where there are some comments from mtmorgan but no other is the issue 611 where he is the reviewer and the submitter of a package.

Bioconductor Bot

We have seen that one of the most relevant “users” is bioc-issue-bot.
It is a bot that performs and report automatic checks on the submissions (The code can be found here). Let’s explore what does and what does it report

Classifying the comments reports that most of the comments are build results or that received a valid push. We can also see some changes on the bot, like changing the messages or reporting differently the process triggered. However, it also reports common problems on the submission: missing repository, unmatch between the repository name and the package name, reposting the same package, missing SSH key (needed to be able to push to Bioconductor git server), … We can see that most of them are related to building the packages being submitted.

Let’s check if there is some differences on the comments according to if they are later accepted or not.

We can see clearly differences on the behavior of the issues, some type of comments are all of the non-accepted packages. It seems like some are not corrected. We can see here some common areas when submissions fail: failing to provide a valid link to the repository or many links to several repositories, or failing to comply with the guidelines about the repository name and the package name. Reposting the same package is also a common reason of not getting that submission approved.

The only significant are the build results, the build results, the valid push, received, unmatch between the package name and the repository, and the update version.

We can see, (and I expected so) that comments of bioc-issue-bot are driven by the build system.

But the other comments are more frequent among the not approved packages, mainly errors and other automatically detected problems.

Aside from the bot, other users might comment on issues too:

In some issues reviewers comment more than authors, while on some there are more comments from authors than reviewers. Surprisingly, in some issues there are more comments from other users than authors or reviewers.

Usually packages not approved have less comments from authors, reviewers and other users.

About the time to get feedback we have previously seen that most of the issues were fast. However it included automatic actions from the bot, so let’s check again without the automatic events.

Here the squares indicate when a issue is closed, and we can see most comments are before the issues are closed.

And there doesn’t seem to be any difference between approved and not approved packages.

I would have expected more comments after closing an issue in the not approved submissions. This might indicate that the discussion happens before and/or that the process after closing the issue is not clear enough.

Steps

So far we have compared closed and open issues, users, comments, events, but we haven’t looked how does the proccess goes.

Let’s see how much time does it take to start the review and the time till the reviewer comments:

Successful build?	Approved?	Submissions
No	No	608
Yes	No	78
Yes	Ongoing	28
No	Ongoing	19
Yes	Yes	723
No	Yes	125

We can see that most not approved packages do not have a successful build.

An overview of the submissions, when was the first successful build and when was the first comment of the reviewer after the successful build.

Usually it takes close to 3 days to have the first successful build on Bioconductor (if there is any as we have seen).

After it it takes close to 10 days to the reviewer to comment.

This comment might be the review of the package, as it is only done after the submissions passes all the checks.

A we have seen on the first table the main blocking point is that many submitted packages do not successfully build on Bioconductor servers.
But as we can see approved packages built slightly earlier. However the time between the first successful built and the first reviewer comment might be different, let’s check:

We can see there aren’t many differences on the time between the first successful built and the first reviewer comment. Once a package has built and has the review it takes some time to address the comments till it is accepted:

We can see that since submission it usually takes two weeks to have the package accepted. However, the time between the first reviewer comment (after the first successful built) and the acceptance of the package might be different:

We can see that few packages take more than 50 days till acceptance. To have a complete picture we can see all the issues on a single plot with how much do they take to move to the next phase:

We can see odd packages like the one that built correctly in short time but had some issues and finally got renamed and submitted on a different issue, or the issue that took more than 100 days to build correctly for the first time (and was later rejected).

We can see that the steps that take more time is doing the review and later modify the package to address the comments pointed by the reviewers.

Approved	Succcessful built (days)	First reviewer comment after build (days)	Accepted (days)	Time between build and reviewer comment (days)	Time between review and acceptance (days)
No	5	19	NA	14	NA
Ongoing	3	25	NA	19	NA
Yes	3	14	37	10	19

Looking at the median times of each step we can clearly see the same pattern.

Labels

So far we have looked via a combination of comments from reviewers and labels. However, the official way of showing on which step is a review is using labels.

We have seen is the second most frequent event, around 1100 submissions have at least one label annotation. We can see here how many:

We can see that the version bump, and the state of the build are the most frequent labels.

Looking into each label we can see differences between acceoted and declined submissions:

Here we can see the differences in label assignment on each issue (labels which are assigned only to approved or rejected packages are not shown).

Some submissions were initially declined but later got approved, as well as some issues that for some time went inactive but ended up being accepted.

The main difference is how many times there are errors, timeouts, warnings, version bump required or OK on the issues approved. Successful packages have more! If you submit a package and get those errors don’t worry, it is normal!!

Approved	VERSION BUMP REQUIRED	ERROR	WARNINGS	OK	TIMEOUT	ABNORMAL	1. awaiting moderation	3a. accepted	2. review in progress	3c. inactive	3b. declined
No	0	1	0	0	0	0	1	0	1	0	0
Yes	2	2	1	2	0	0	1	1	1	0	0

We can see that is more common to have more “troubles” on packages that end up accepted than those than not. Note that there are at least two OK per submission, one required before the review start and the other one after the changes before acceptance.

Conclusions

Most of the problems with the submissions are formal and automatically detected by the bot. Next come problems from the package itself not passing the checks performed on Bioconductor. So if you want to have your package included make sure that the package builds on Bioconductor and respond fast to the feedback provided by the bot. Once your package successfully builds, then address the comments from the reviewer. Follow the steps and if you don’t drop out you’ll see your package accepted.

On the next post I’ll explore the data from rOpenSci, which also does the reviews on GitHub.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.1 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2021-01-08                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package      * version date       lib source                           
##  assertthat     0.2.1   2019-03-21 [1] CRAN (R 4.0.1)                   
##  backports      1.2.1   2020-12-09 [1] CRAN (R 4.0.1)                   
##  blogdown       0.21.84 2021-01-07 [1] Github (rstudio/blogdown@c4fbb58)
##  bookdown       0.21    2020-10-13 [1] CRAN (R 4.0.1)                   
##  broom          0.7.3   2020-12-16 [1] CRAN (R 4.0.1)                   
##  cellranger     1.1.0   2016-07-27 [1] CRAN (R 4.0.1)                   
##  cli            2.2.0   2020-11-20 [1] CRAN (R 4.0.1)                   
##  codetools      0.2-18  2020-11-04 [1] CRAN (R 4.0.1)                   
##  colorspace     2.0-0   2020-11-11 [1] CRAN (R 4.0.1)                   
##  crayon         1.3.4   2017-09-16 [1] CRAN (R 4.0.1)                   
##  DBI            1.1.0   2019-12-15 [1] CRAN (R 4.0.1)                   
##  dbplyr         2.0.0   2020-11-03 [1] CRAN (R 4.0.1)                   
##  digest         0.6.27  2020-10-24 [1] CRAN (R 4.0.1)                   
##  dplyr        * 1.0.2   2020-08-18 [1] CRAN (R 4.0.1)                   
##  ellipsis       0.3.1   2020-05-15 [1] CRAN (R 4.0.1)                   
##  evaluate       0.14    2019-05-28 [1] CRAN (R 4.0.1)                   
##  fansi          0.4.1   2020-01-08 [1] CRAN (R 4.0.1)                   
##  farver         2.0.3   2020-01-16 [1] CRAN (R 4.0.1)                   
##  forcats      * 0.5.0   2020-03-01 [1] CRAN (R 4.0.1)                   
##  fs             1.5.0   2020-07-31 [1] CRAN (R 4.0.1)                   
##  generics       0.1.0   2020-10-31 [1] CRAN (R 4.0.1)                   
##  ggplot2      * 3.3.3   2020-12-30 [1] CRAN (R 4.0.1)                   
##  ggrepel      * 0.9.0   2020-12-16 [1] CRAN (R 4.0.1)                   
##  gh             1.2.0   2020-11-27 [1] CRAN (R 4.0.1)                   
##  glue           1.4.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  gtable         0.3.0   2019-03-25 [1] CRAN (R 4.0.1)                   
##  haven          2.3.1   2020-06-01 [1] CRAN (R 4.0.1)                   
##  here           1.0.1   2020-12-13 [1] CRAN (R 4.0.1)                   
##  highr          0.8     2019-03-20 [1] CRAN (R 4.0.1)                   
##  hms            0.5.3   2020-01-08 [1] CRAN (R 4.0.1)                   
##  htmltools      0.5.0   2020-06-16 [1] CRAN (R 4.0.1)                   
##  httr           1.4.2   2020-07-20 [1] CRAN (R 4.0.1)                   
##  jsonlite       1.7.2   2020-12-09 [1] CRAN (R 4.0.1)                   
##  knitr          1.30    2020-09-22 [1] CRAN (R 4.0.1)                   
##  labeling       0.4.2   2020-10-20 [1] CRAN (R 4.0.1)                   
##  lifecycle      0.2.0   2020-03-06 [1] CRAN (R 4.0.1)                   
##  lubridate      1.7.9.2 2020-11-13 [1] CRAN (R 4.0.1)                   
##  magrittr       2.0.1   2020-11-17 [1] CRAN (R 4.0.1)                   
##  modelr         0.1.8   2020-05-19 [1] CRAN (R 4.0.1)                   
##  munsell        0.5.0   2018-06-12 [1] CRAN (R 4.0.1)                   
##  patchwork    * 1.1.1   2020-12-17 [1] CRAN (R 4.0.1)                   
##  pillar         1.4.7   2020-11-20 [1] CRAN (R 4.0.1)                   
##  pkgconfig      2.0.3   2019-09-22 [1] CRAN (R 4.0.1)                   
##  purrr        * 0.3.4   2020-04-17 [1] CRAN (R 4.0.1)                   
##  R6             2.5.0   2020-10-28 [1] CRAN (R 4.0.1)                   
##  RColorBrewer   1.1-2   2014-12-07 [1] CRAN (R 4.0.1)                   
##  Rcpp           1.0.5   2020-07-06 [1] CRAN (R 4.0.1)                   
##  readr        * 1.4.0   2020-10-05 [1] CRAN (R 4.0.1)                   
##  readxl         1.3.1   2019-03-13 [1] CRAN (R 4.0.1)                   
##  reprex         0.3.0   2019-05-16 [1] CRAN (R 4.0.1)                   
##  rlang          0.4.10  2020-12-30 [1] CRAN (R 4.0.1)                   
##  rmarkdown      2.6     2020-12-14 [1] CRAN (R 4.0.1)                   
##  rprojroot      2.0.2   2020-11-15 [1] CRAN (R 4.0.1)                   
##  rstudioapi     0.13    2020-11-12 [1] CRAN (R 4.0.1)                   
##  rvest          0.3.6   2020-07-25 [1] CRAN (R 4.0.1)                   
##  scales         1.1.1   2020-05-11 [1] CRAN (R 4.0.1)                   
##  sessioninfo    1.1.1   2018-11-05 [1] CRAN (R 4.0.1)                   
##  socialGH     * 0.0.3   2020-08-17 [1] local                            
##  stringi        1.5.3   2020-09-09 [1] CRAN (R 4.0.1)                   
##  stringr      * 1.4.0   2019-02-10 [1] CRAN (R 4.0.1)                   
##  tibble       * 3.0.4   2020-10-12 [1] CRAN (R 4.0.1)                   
##  tidyr        * 1.1.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  tidyselect     1.1.0   2020-05-11 [1] CRAN (R 4.0.1)                   
##  tidyverse    * 1.3.0   2019-11-21 [1] CRAN (R 4.0.1)                   
##  vctrs          0.3.6   2020-12-17 [1] CRAN (R 4.0.1)                   
##  viridisLite    0.3.0   2018-02-01 [1] CRAN (R 4.0.1)                   
##  withr          2.3.0   2020-09-22 [1] CRAN (R 4.0.1)                   
##  xfun           0.20    2021-01-06 [1] CRAN (R 4.0.1)                   
##  xml2           1.3.2   2020-04-23 [1] CRAN (R 4.0.1)                   
##  yaml           2.2.1   2020-02-01 [1] CRAN (R 4.0.1)                   
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library