Bioconductor | B101nfo

Packaging R: getting in repositories

Sun, 05 May 2024 00:00:00 +0000

After the previous post collecting information about repositories I want to collect here my thoughts on adding a package in a repository and how repositories are recognized. As in the previous post this is built on the assumption that one already has a package or more and wants to distribute it.

This is meant as a reflection of what is an R repository and not intended for R package developers. However, their feedback is appreciated to consider how an ideal repository would be.

Package submission

An R repository will have a way to incorporate a package. CRAN submission process starts with a form, while Bioconductor is done through a Github issue.

The process will usually then start with an automated process. Until the automated process check hasn’t passed probably no one will look into the package submission. This reduce the hours a human must dedicate to manage submissions. If a man is kept in the loop one could appeal the automatic process contacting them, or if it is a random failing re-submitting the package again.

Package submission checks: first a check of the package, if it is not new a dependency check from the repository if all checks pass the package is added to the repository.

Generally a package must first pass a package quality check before it is considered for further integration test. This integration test is usually checking the new version of a package with packages that depend on it, also known as reverse dependencies.

Package maintenance

Once a package is included in a repository it usually needs to be maintained.

There are many moving pieces, chips architecture, OS, R, other packages. This all lead that authors need to maintain the packages in good shape if they want it to remain useful to users. Of course, if one doesn’t want to do that they do not need to create a repository to share their package.

Graphic showing time and different R versions and checks. Repositories check the packages on them on multiple levels.

This leads that at any given time point there must be some tests for any given package under different conditions as shown in image 2. This leads to the possibility of having a package archived from the repository for failing the checks in place.

Repositories provide these checks as a service to the users. They guarantee that R packages in the repository work well together and pass the same set of packages (mostly). This is what leads to their reputation and usage among users (this is true beyond R, DEBIAN, Ubuntu, …).

Closing remarks

There are several official repositories how the package submission works when a package is submitted to one but it is related, via dependencies to other repositories is a matter of another post.

There are some discussion on what is an R repository. The importance of CRAN and Bioconductor has lead to some confusion. There are generally two meanings of what a cran-like repository is:

One where install.packages() works (This is defined by how the files and binaries are organized and will be a theme for another time).
One were all the checks described here are in place and install.packages() works too.

r-universe is using the first definition but could be used to generate repositories with checks that comply with the second definition. Other repositories that use that are the Posit Public Package Manager, or the R4Pi repository (which provides binaries for Raspberry Pi OS).

As the second definition is more strict I’ll focus on it as this post has explained.

PS: This post might be edited as it has been siting in my computer for several months. I prefer to post it and be improved with feedback, so let me know if you have any addition.

Packaging R: repositories

Sat, 09 Dec 2023 00:00:00 +0000

In this post I want to collect some thoughts about R repositories. In R we have multiple repositories that store packages for users. In this post I want to write about the purpose, functionality, benefits and drawbacks of R repositories and how packages are managed. The goal is to summarize what I’ve learnt these last years about them. I’ll also collect some information about them from various sources to make it easier for myself to find it later on.

I am writing this because I am worried about the future of CRAN and R. Due to multiple circumstances, the current position is not sustainable as is. I hope that this post, will help me to understand the past, present and create some concrete steps to do.

History

I was not there, but the first repository started around April 1997. This repository is CRAN: the Comprehensive R Archive Network. The first mention I found is already about changes in it, but it was not until the end of the month when it was announced.

CRAN was created by a few volunteers, some of which are still mainting it 25 years later. The current team is listed on their website. From the beginning it was “a collection of sites which carry identical material, consisting of the R&R R distribution(s), the contributed extensions, documentation for R, and binaries.”

Omegahat was another repository created shortly after CRAN:

The Omega project began in July, 1998, with discussions among designers responsible for three current statistical languages (S, R, and Lisp-Stat), with the idea of working together on new directions with special emphasis on web-based software, Java, the Java virtual machine, and distributed computing.

Many developers of Omegahat were in the R Core or CRAN team. It was available as a repository from the R source code but was removed definitely in version R 4.1, in 2021¹.

Bioconductor, was the next major repository that appeared. It was funded by Robert Gentleman and others in 2004 (it started the mailing list). A paper describing it appeared in late 2004:

an initiative for the collaborative creation of extensible software for computational biology and bioinformatics.

Through its history repositories have evolved with R and R with them. For example: R was released twice a year at the beginning, and Bioconductor did too. But when R moved to be released once per year (in 2013 with version 3.0) Bioconductor kept using two releases a year. This introduced some problems when installing packages from Bioconductor, when a single R release can be compatible with two Bioconductor releases².

In other cases, checks have evolved. For instance Solaris was used to test packages in CRAN until 2021, if I recall correctly, because it allowed to test in a proprietary C or C++ compiler. This lead to discover bugs but also to more distress in R-package developers which had difficulties checking their packages in that environment.

Other checks evolve with R, becoming more strict with time: In the early versions of R the use of NAMESPACE was not regulated. But since R version 2.15, 2012 it was compulsory even for data-only packages³. This was synchronized with repositories checks.

Last, some goals/desires of CRAN are not fulfilled (or where abandoned). For example, from the start CRAN aimed to have packages authenticated (see the bottom of the announcement). This might be due to lack of time, resources or that the plans are in progress but require (volunteer) time.

With time, different repositories arose:

MRAN, which was available since September 17th, 2014 to July 1st, 2022.
The Rstudio Public Package Manager later renamed Posit Public Package Manager has binaries for several OS since 2019.
There is the R4pi repository with binaries for Raspberry Pi.
I remember a proteomics repository available.
rOpenSci started its own repository which later evolved into the r-universe. The r-universe currently can provide binaries of packages that are hosted in a git repository.

Literature

The role and prominence of the repositories has lead to many articles being written about it. I wanted to link and collect some of them for easier retrieval.

I was wondering how CRAN is described by the volunteers that built it. From the announcing email:

CRAN is a collection of sites which carry identical material, consisting of the R&R R distribution(s), the contributed extensions, documentation for R, and binaries.

From the website (at 2023/12/09):

CRAN is a network of ftp and web servers around the world that store identical, up-to-date, versions of code and documentation for R.

Initially there was R NEWS, with an article dedicated to CRAN and one to Omegahat too. These articles usually describe new package additions but sometimes they also provide information about changes:

CRAN-2001-1: It list new packages, CRAN-2001-2, CRAN-2001-3, Omegahat-2001-3
CRAN-2002-1, CRAN-2002-2, CRAN-2002-3
CRAN-2003-1, CRAN-2003-2, CRAN-2003-3
CRAN-2004-1, CRAN-2004-2
CRAN-2005-1, CRAN-2005-2

Since 2006 there is also an article about Bioconductor.
CRAN-2006-2, Bioc-2006-2
CRAN-2007-1, CRAN-2007-2, Bioc-2007-2, CRAN-2007-3
CRAN-2008-1, Bioc-2008-1 CRAN-2008-2, Bioc-2008-2

Later it became the R Journal:

CRAN-2009-1, CRAN-2009-2
CRAN-2010-1, CRAN-2010-2
CRAN-2011-1, CRAN and Bioconductor 2011-2. In the bioconductor section it mentions conference, and important directions for the Bioconductor core.
CRAN-2012-1, CRAN and Bioconductor 2012-2: Mentions biocLite() to install packages.
CRAN-2013-1 Bioc-2013-1: mentions better integration of parallel evaluation.
CRAN-2013-2, Bioc-2013-2: Mentions again AnnotationHub
CRAN-2014-1, Bioc-2014-1: Mentions the git-svn bridge to synchronize git and svn repository.
CRAN-2014-2, Bioc-2014-2: Bioconductor 3.0 release, besides some packages Amazon Machine Image are offered as well as docker images. Packages are required to pass BiocCheck, checks in a different package specific for Bioconductor.
CRAN-2015-1, Bioc-2015-1: Same mentions as the previous and encouragement to guidelines an package submission.
CRAN-2015-2, Bioc-2015-2
CRAN-2016-1: on this article there is a plot of the number of CRAN packages and time, and doesn’t list all packages listed. It explicitly mentions that the CRAN team asked for help processing package submissions and some people stepped up. Bioc-2016-1

CRAN-2016-2, Bioc-2016-2
CRAN-2017-1: mentions changes in CRAN checks, adding new memory access and static code analysis checks. It mentions that the submission has moved to a more automated one. It also mentions changes in the CRAN Repository Policy. Bioc-2017-1
CRAN-2018-1: checks in alternative BLAS/LAPACK implementations, the submission pipeline is defined. First time the amount of action taken by CRAN reviewers is listed in two categories automatic and manual. Changes in repository policy are listed. Changes in location of package repository archive , Bioc-2018-1
CRAN-2018-2: Changes in policy; packages should not give a check warning nor error. Bioc-2018-2: Moved to BiocManager to install packages.
CRAN-2019-1: More mentions to CRAN mirror security.

CRAN-2019-2: Updates in checklist for CRAN submissions, Bioc-2019-2
CRAN-2020-1: Many changes in CRAN policies. CRAN-2020-2: Many changes to CRAN policies. Bioc-2020-2: Announces the Technical and Community advisory boards (as well as the project-wide Code of Conduct).
CRAN-2021-1, Bioc-2021-1: Mentions conferences that will be virtual.
CRAN-2021-2: Shows an incomig path [See this friendly viewer, Bioc-2021-2: Mentions AnVIL and two online workshops to develop workflows.
CRAN-2022-1: List a change in CRAN policy and the CRAN Task View Initiative.
CRAN-2022-2: List some more repository policies. Bioc-2022-2: Lists infrastructure updates (and its funding), changes in the core team and new initiatives.
CRAN-2022-3, Bioc-2022-3
CRAN-2022-4, Bioc-2022-4: default branch renaming, partnership with Outreachy and blog are featured. Several working groups provide updates
CRAN-2023-1

In addition, several articles and blog posts have appeared. From those I found it is worth mentioning the following:

And my own posts:

Reasons CRAN packages are archived
CRAN files part 1
CRAN files part 2
CRAN maintained packages
CRAN review (and the talk at useRs 2021)
Bioconductor review
rOpenSci reviews
Bioconductor reviews
The article “Aspects of the Social Organization and Trajectory of the R Project”, from the R Journal 2009, also has a section about CRAN, noting that it “is challenged by its own success”.

Characteristics

The predominance of CRAN and its role as primary and default R repository has lead to some special treatment of the repository.

CRAN checks are in the R source code itself. While other repositories have their own checks in different tools. In addition, the CRAN environmental variables used are documented in the R-internals (they are more or less accessible in the svn repository too).

Others who know more have stated the benefits of CRAN too: This text is copied from Henrik Bengstsson in Bioconductor Slack:

FOREVER ARCHIVE:

The first one is that it publishes packages and versions of them until the end of time. When a package has been published on CRAN, it takes a lot for it to be removed from there. I don’t know if it ever happened, but I can imagine a package can be fully removed if it was illegally published in the first place (e.g. copyright, illegal content, ...) or malicious.

INSTALLATION SERVICE:

Then CRAN also provides a R package repository service for installing packages on CRAN using built-in R functions. The set of packages in the package repo is a subset of all packages on CRAN. The CRAN package repo makes a promise that all packages listed in PACKAGES can be installed. If they cannot make that promise, they’ll archive the package (=remove it from PACKAGES). I should also say, install.packages(url) can be used to install from the set of packages that are archived. Technically, old package versions are always archived.

CHECK SERVICE:

The content of the R package repository is guided by the CRAN package checks that run on R-oldrel, R-release, and R-devel across multiple platforms. The minimal requirement is that no package should remain in the package repository if the checks detects ERRORs (and those errors are not due to recently introduced bugs in R-devel). WARNINGs can also cause a package to be archived, but that process often takes longer. AFAIK, NOTEs are not a cause for a package being archived (but I could be wrong). The CRAN incoming checks, which you have to pass when you submit a new package, or an updated version, will make sure that the published package pass with all OKs. (It’s possible to argue for NOTEs being false positives, or for them not to be fixed, but that requires a manual approval by the CRAN Team).

I think there are many more resources discussing R repositories. If you know more I’ll be happy to update this post.

I think before I drag too much on the steps I’ll post this and collect some more articles I might have missed.

Last, Uwe Liegges presented about CRAN in useR!2017, thanks Tim Taylor for sharing it. In this video there is an explanation of why the solaris OS was used.

It has come to my attention that there is an article, by G. Brooke Anderson and Dirk Eddelbuette, about the R package repositories structure (among other things): Hosting Data Packages via drat: A Case Study with Hurricane Exposure Data

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       Ubuntu 22.04.3 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2024-01-15
##  pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date (UTC) lib source
##  blogdown      1.18    2023-06-19 [1] CRAN (R 4.3.1)
##  bookdown      0.37    2023-12-01 [1] CRAN (R 4.3.1)
##  bslib         0.6.1   2023-11-28 [1] CRAN (R 4.3.1)
##  cachem        1.0.8   2023-05-01 [1] CRAN (R 4.3.1)
##  cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.1)
##  digest        0.6.33  2023-07-07 [1] CRAN (R 4.3.1)
##  evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.2)
##  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.1)
##  htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.2)
##  jquerylib     0.1.4   2021-04-26 [1] CRAN (R 4.3.1)
##  jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.3.1)
##  knitr         1.45    2023-10-30 [1] CRAN (R 4.3.2)
##  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.2)
##  R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.1)
##  rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.1)
##  rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.1)
##  rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
##  sass          0.4.8   2023-12-06 [1] CRAN (R 4.3.1)
##  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.1)
##  xfun          0.41    2023-11-01 [1] CRAN (R 4.3.2)
##  yaml          2.3.8   2023-12-11 [1] CRAN (R 4.3.1)
## 
##  [1] /home/lluis/bin/R/4.3.1
##  [2] /opt/R/4.3.1/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

In version 3.1.2 Omegahat didn’t provide Windows binaries and in 4.1 from the default repositories (See 4.1 in NEWS(.4)).↩︎
This lead to the need of having a special function to install packages from Bioconductor. Initially a function biocLite and later with the BiocManager package.↩︎
NEWS in 2.15 section ↩︎

Bioconductor submissions: reviews

Fri, 31 Jul 2020 00:00:00 +0000

First post is on Bioconductor submissions raised some questions comments but at the time I didn’t have a good way to answer them:

issues get closed after they got assigned a reviewer and before the reviewer actually gets a chance to start the review.
issues assigned to multiple people or issues that switched reviewers

To answer both of them I needed more information. On the previous post I only gathered the information of the state of the issues at that moment. This excluded label changes, reviewer changes, renaming the issues, who commented on the issues and many more things. To retrieve these information from github I developed a new package {socialGH} which downloads it from Github to make this analysis possible.

I gathered all the information on issues using {socialGH} on the 2020/08/18 and stored them in a tidy format. Now that we have more data about the issues we can plot them similarly to what we did on the previous post:

We can see that sometimes the issues remained silent for several months and then had a high level of events, or a single one (closing mainly).

Compare to the previous version it is surprising to see that one of the earliest issues still gets new events to date. Apparently issue 51, and others, are being used to test the Bioconductor builder or the bot used to automate the process.

We can see that some reviewers comment more and are involved in more issues, but there is a group of fairly similar comments and issues.

Clearly this will not be the same with users who submit packages:

We can clearly see that most users are mainly only involved on submitting packages on one issue, but some people participate a lot and/or in several issues.

Events

We cab count how many users and issues have been involved with each event type:

We can see that few issues are locked (remember that it is different than closing the issue) or comments deleted. There is an almost equal amount of labeling and unlabeling, performed by few people but on lots of issues. There is a group of mentioning, commenting and subscribing too. Many people is involved on commenting and creating submissions.

Looking a bit further on the issues we can look at the events that take place:

We can see that most issues have few events which agrees with the previous findings that issues are handled fairly and expeditiously.

The most common event are comments, mentions and the rarest events are deleting comments, locking or referencing them.

We can see that most issues get around 50 events, but we can clearly see lots of issues that receive very few events.

However, looking at the distribution along time most of the events are on the first 6 months after opening the issue.

Some submissions have many events while others have few events but last longer. This is without looking at issues that were closed and later reopened, or the involvement of bioc-issue-bot. We can see that in a short amount of days a lot of events can be triggered. This is mostly that for each version update there are at least two messages on the issue.

Reviews

Most of the differences are due to the automatic or manual review of the submissions.

After passing some preliminary checks a reviewer is assigned to manually review the package:

We can see that usually some adjustment to reviewers is made usually changing them but some times they are added to have more than 1 reviewer (194). Almost all the submitted packages without a reviewer (537 submissions) were rejected except for three: 81, 82, 83 that were reviewed by vobencha even if not officially assigned. Some of these were rejected because they didn’t pass some automatic check and other after preliminary inspection.

If some of these preliminary checks aren’t corrected the issue is closed:

We can see that is common to close and reopen issues, but they are generally done by the same people (the reviewer, the original submitter or the bot).

On submissions with a reviewer assigned that were closed by someone (not the submitter), we can see here that reviews with more than one reviewer do not lead to significantly more comments from the reviewers.

Although it seems like this leads to have the package approved.

We can see that there are slightly more comments from reviewers on the approved packages. Perhaps because they provide more feedback once the review starts.

We can see that the percentage of approved on those issues where the reviewer commented is fairly similar and around 80% or above. However there is also at least a 5% of submissions that are closed despite the comments from reviewers. Usually this has to be with unresponsiveness from the submitter:

We can see that on the approved packages there are usually more events on the same time period. So make sure to follow the advice of the reviewers and the bot and make all the errors and warnings disappear.

While most issues have at least 8 different events they usually have 4 users involved. Presumably the creator of the issue, bioc-issue-bot, a reviewer and someone else.

The more users involved, more different type of events are triggered. Presumably more people get subscribed or is mentioned.

As expected the more users are involved in an issue more events are produced. We can also see the submissions that are closed by the bioc-issue-bot with few comments and users.

Who does each action ?

We can look now at who performs what, we know there are 951 participants:

Here I cut the top 35 people who have triggered more than events. We can clearly see who are official reviewers (as seen on the previous post) because they labeled and unlabeled issues. bioc-issue-bot is an special user that makes lots of comments and assigning reviewers to issues. We can also see that it previously renamed the issues or unassigned reviewers.

Comments by users (excluding the bot), usually only the author, the reviewer comment, frequently mtmorgan also suggests.

We can see that Martin Morgan has commented on almost all the submissions. The odd issue where there are some comments from mtmorgan but no other is the issue 611 where he is the reviewer and the submitter of a package.

Bioconductor Bot

We have seen that one of the most relevant “users” is bioc-issue-bot.
It is a bot that performs and report automatic checks on the submissions (The code can be found here). Let’s explore what does and what does it report

Classifying the comments reports that most of the comments are build results or that received a valid push. We can also see some changes on the bot, like changing the messages or reporting differently the process triggered. However, it also reports common problems on the submission: missing repository, unmatch between the repository name and the package name, reposting the same package, missing SSH key (needed to be able to push to Bioconductor git server), … We can see that most of them are related to building the packages being submitted.

Let’s check if there is some differences on the comments according to if they are later accepted or not.

We can see clearly differences on the behavior of the issues, some type of comments are all of the non-accepted packages. It seems like some are not corrected. We can see here some common areas when submissions fail: failing to provide a valid link to the repository or many links to several repositories, or failing to comply with the guidelines about the repository name and the package name. Reposting the same package is also a common reason of not getting that submission approved.

The only significant are the build results, the build results, the valid push, received, unmatch between the package name and the repository, and the update version.

We can see, (and I expected so) that comments of bioc-issue-bot are driven by the build system.

But the other comments are more frequent among the not approved packages, mainly errors and other automatically detected problems.

Aside from the bot, other users might comment on issues too:

In some issues reviewers comment more than authors, while on some there are more comments from authors than reviewers. Surprisingly, in some issues there are more comments from other users than authors or reviewers.

Usually packages not approved have less comments from authors, reviewers and other users.

About the time to get feedback we have previously seen that most of the issues were fast. However it included automatic actions from the bot, so let’s check again without the automatic events.

Here the squares indicate when a issue is closed, and we can see most comments are before the issues are closed.

And there doesn’t seem to be any difference between approved and not approved packages.

I would have expected more comments after closing an issue in the not approved submissions. This might indicate that the discussion happens before and/or that the process after closing the issue is not clear enough.

Steps

So far we have compared closed and open issues, users, comments, events, but we haven’t looked how does the proccess goes.

Let’s see how much time does it take to start the review and the time till the reviewer comments:

Successful build?	Approved?	Submissions
No	No	608
Yes	No	78
Yes	Ongoing	28
No	Ongoing	19
Yes	Yes	723
No	Yes	125

We can see that most not approved packages do not have a successful build.

An overview of the submissions, when was the first successful build and when was the first comment of the reviewer after the successful build.

Usually it takes close to 3 days to have the first successful build on Bioconductor (if there is any as we have seen).

After it it takes close to 10 days to the reviewer to comment.

This comment might be the review of the package, as it is only done after the submissions passes all the checks.

A we have seen on the first table the main blocking point is that many submitted packages do not successfully build on Bioconductor servers.
But as we can see approved packages built slightly earlier. However the time between the first successful built and the first reviewer comment might be different, let’s check:

We can see there aren’t many differences on the time between the first successful built and the first reviewer comment. Once a package has built and has the review it takes some time to address the comments till it is accepted:

We can see that since submission it usually takes two weeks to have the package accepted. However, the time between the first reviewer comment (after the first successful built) and the acceptance of the package might be different:

We can see that few packages take more than 50 days till acceptance. To have a complete picture we can see all the issues on a single plot with how much do they take to move to the next phase:

We can see odd packages like the one that built correctly in short time but had some issues and finally got renamed and submitted on a different issue, or the issue that took more than 100 days to build correctly for the first time (and was later rejected).

We can see that the steps that take more time is doing the review and later modify the package to address the comments pointed by the reviewers.

Approved	Succcessful built (days)	First reviewer comment after build (days)	Accepted (days)	Time between build and reviewer comment (days)	Time between review and acceptance (days)
No	5	19	NA	14	NA
Ongoing	3	25	NA	19	NA
Yes	3	14	37	10	19

Looking at the median times of each step we can clearly see the same pattern.

Labels

So far we have looked via a combination of comments from reviewers and labels. However, the official way of showing on which step is a review is using labels.

We have seen is the second most frequent event, around 1100 submissions have at least one label annotation. We can see here how many:

We can see that the version bump, and the state of the build are the most frequent labels.

Looking into each label we can see differences between acceoted and declined submissions:

Here we can see the differences in label assignment on each issue (labels which are assigned only to approved or rejected packages are not shown).

Some submissions were initially declined but later got approved, as well as some issues that for some time went inactive but ended up being accepted.

The main difference is how many times there are errors, timeouts, warnings, version bump required or OK on the issues approved. Successful packages have more! If you submit a package and get those errors don’t worry, it is normal!!

Approved	VERSION BUMP REQUIRED	ERROR	WARNINGS	OK	TIMEOUT	ABNORMAL	1. awaiting moderation	3a. accepted	2. review in progress	3c. inactive	3b. declined
No	0	1	0	0	0	0	1	0	1	0	0
Yes	2	2	1	2	0	0	1	1	1	0	0

We can see that is more common to have more “troubles” on packages that end up accepted than those than not. Note that there are at least two OK per submission, one required before the review start and the other one after the changes before acceptance.

Conclusions

Most of the problems with the submissions are formal and automatically detected by the bot. Next come problems from the package itself not passing the checks performed on Bioconductor. So if you want to have your package included make sure that the package builds on Bioconductor and respond fast to the feedback provided by the bot. Once your package successfully builds, then address the comments from the reviewer. Follow the steps and if you don’t drop out you’ll see your package accepted.

On the next post I’ll explore the data from rOpenSci, which also does the reviews on GitHub.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.1 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2021-01-08                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package      * version date       lib source                           
##  assertthat     0.2.1   2019-03-21 [1] CRAN (R 4.0.1)                   
##  backports      1.2.1   2020-12-09 [1] CRAN (R 4.0.1)                   
##  blogdown       0.21.84 2021-01-07 [1] Github (rstudio/blogdown@c4fbb58)
##  bookdown       0.21    2020-10-13 [1] CRAN (R 4.0.1)                   
##  broom          0.7.3   2020-12-16 [1] CRAN (R 4.0.1)                   
##  cellranger     1.1.0   2016-07-27 [1] CRAN (R 4.0.1)                   
##  cli            2.2.0   2020-11-20 [1] CRAN (R 4.0.1)                   
##  codetools      0.2-18  2020-11-04 [1] CRAN (R 4.0.1)                   
##  colorspace     2.0-0   2020-11-11 [1] CRAN (R 4.0.1)                   
##  crayon         1.3.4   2017-09-16 [1] CRAN (R 4.0.1)                   
##  DBI            1.1.0   2019-12-15 [1] CRAN (R 4.0.1)                   
##  dbplyr         2.0.0   2020-11-03 [1] CRAN (R 4.0.1)                   
##  digest         0.6.27  2020-10-24 [1] CRAN (R 4.0.1)                   
##  dplyr        * 1.0.2   2020-08-18 [1] CRAN (R 4.0.1)                   
##  ellipsis       0.3.1   2020-05-15 [1] CRAN (R 4.0.1)                   
##  evaluate       0.14    2019-05-28 [1] CRAN (R 4.0.1)                   
##  fansi          0.4.1   2020-01-08 [1] CRAN (R 4.0.1)                   
##  farver         2.0.3   2020-01-16 [1] CRAN (R 4.0.1)                   
##  forcats      * 0.5.0   2020-03-01 [1] CRAN (R 4.0.1)                   
##  fs             1.5.0   2020-07-31 [1] CRAN (R 4.0.1)                   
##  generics       0.1.0   2020-10-31 [1] CRAN (R 4.0.1)                   
##  ggplot2      * 3.3.3   2020-12-30 [1] CRAN (R 4.0.1)                   
##  ggrepel      * 0.9.0   2020-12-16 [1] CRAN (R 4.0.1)                   
##  gh             1.2.0   2020-11-27 [1] CRAN (R 4.0.1)                   
##  glue           1.4.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  gtable         0.3.0   2019-03-25 [1] CRAN (R 4.0.1)                   
##  haven          2.3.1   2020-06-01 [1] CRAN (R 4.0.1)                   
##  here           1.0.1   2020-12-13 [1] CRAN (R 4.0.1)                   
##  highr          0.8     2019-03-20 [1] CRAN (R 4.0.1)                   
##  hms            0.5.3   2020-01-08 [1] CRAN (R 4.0.1)                   
##  htmltools      0.5.0   2020-06-16 [1] CRAN (R 4.0.1)                   
##  httr           1.4.2   2020-07-20 [1] CRAN (R 4.0.1)                   
##  jsonlite       1.7.2   2020-12-09 [1] CRAN (R 4.0.1)                   
##  knitr          1.30    2020-09-22 [1] CRAN (R 4.0.1)                   
##  labeling       0.4.2   2020-10-20 [1] CRAN (R 4.0.1)                   
##  lifecycle      0.2.0   2020-03-06 [1] CRAN (R 4.0.1)                   
##  lubridate      1.7.9.2 2020-11-13 [1] CRAN (R 4.0.1)                   
##  magrittr       2.0.1   2020-11-17 [1] CRAN (R 4.0.1)                   
##  modelr         0.1.8   2020-05-19 [1] CRAN (R 4.0.1)                   
##  munsell        0.5.0   2018-06-12 [1] CRAN (R 4.0.1)                   
##  patchwork    * 1.1.1   2020-12-17 [1] CRAN (R 4.0.1)                   
##  pillar         1.4.7   2020-11-20 [1] CRAN (R 4.0.1)                   
##  pkgconfig      2.0.3   2019-09-22 [1] CRAN (R 4.0.1)                   
##  purrr        * 0.3.4   2020-04-17 [1] CRAN (R 4.0.1)                   
##  R6             2.5.0   2020-10-28 [1] CRAN (R 4.0.1)                   
##  RColorBrewer   1.1-2   2014-12-07 [1] CRAN (R 4.0.1)                   
##  Rcpp           1.0.5   2020-07-06 [1] CRAN (R 4.0.1)                   
##  readr        * 1.4.0   2020-10-05 [1] CRAN (R 4.0.1)                   
##  readxl         1.3.1   2019-03-13 [1] CRAN (R 4.0.1)                   
##  reprex         0.3.0   2019-05-16 [1] CRAN (R 4.0.1)                   
##  rlang          0.4.10  2020-12-30 [1] CRAN (R 4.0.1)                   
##  rmarkdown      2.6     2020-12-14 [1] CRAN (R 4.0.1)                   
##  rprojroot      2.0.2   2020-11-15 [1] CRAN (R 4.0.1)                   
##  rstudioapi     0.13    2020-11-12 [1] CRAN (R 4.0.1)                   
##  rvest          0.3.6   2020-07-25 [1] CRAN (R 4.0.1)                   
##  scales         1.1.1   2020-05-11 [1] CRAN (R 4.0.1)                   
##  sessioninfo    1.1.1   2018-11-05 [1] CRAN (R 4.0.1)                   
##  socialGH     * 0.0.3   2020-08-17 [1] local                            
##  stringi        1.5.3   2020-09-09 [1] CRAN (R 4.0.1)                   
##  stringr      * 1.4.0   2019-02-10 [1] CRAN (R 4.0.1)                   
##  tibble       * 3.0.4   2020-10-12 [1] CRAN (R 4.0.1)                   
##  tidyr        * 1.1.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  tidyselect     1.1.0   2020-05-11 [1] CRAN (R 4.0.1)                   
##  tidyverse    * 1.3.0   2019-11-21 [1] CRAN (R 4.0.1)                   
##  vctrs          0.3.6   2020-12-17 [1] CRAN (R 4.0.1)                   
##  viridisLite    0.3.0   2018-02-01 [1] CRAN (R 4.0.1)                   
##  withr          2.3.0   2020-09-22 [1] CRAN (R 4.0.1)                   
##  xfun           0.20    2021-01-06 [1] CRAN (R 4.0.1)                   
##  xml2           1.3.2   2020-04-23 [1] CRAN (R 4.0.1)                   
##  yaml           2.2.1   2020-02-01 [1] CRAN (R 4.0.1)                   
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library

Bioconductor submissions

Mon, 15 Jun 2020 00:00:00 +0000

The other day I was curious about how many submissions are on Bioconductor and how do they work, is there any pattern toward the release cycle or not? In this brief post I’ll explore some data available from the github repository where the submissions are done as issues. I mostly focus on what happens when one submits a package so that it can help future submissions.

There are around 1826 issues, I extract the interesting information: date of creation, who creates them, where is the linked package. when are they closed, labels at the moment, reviewers…

We can further mark those approved because they are on bioconductor but do not have the approved label (Sometimes they get accepted but they forget to change labels on the issue).

I didn’t mark as accepted packages that were submitted several times (184 packages have been submitted more than once). Some of them are due to being on the old tracker and were not approved on this repository.
Using also the CRAN repository we could find which packages where submitted to Bioconductor but end up on CRAN. But as I expect a low number of these I won’t check them.

Exploring the submissions

First of all a general overview of the data available:

On this plot we can see almost everything, date of creation, time opened, when was closed, in which release is included if accepted and the rate of submissions to Bioconductor. What we miss is about the authors submitting the packages and about the packages themselves.

Submitting authors

One of the core strength of R is the open community. There have been 749 different users submitting at least one package.

We can clearly see that most authors submit just one package but some submit a few more, with an outlier who have submitted more than 10 packages.

Most of the authors get their submission included at Bioconductor at the first try. Few need two submissions and some do not get their package included on Bioconductor. This can be because they withdraw the submission, or they decide to submit elsewhere or simply they do not go through the review.

If authors submit more packages they usually get them approved. Although we can also see some users that make several submissions.

Regardless if the package is submitted just once or several times. More submissions do not make the package get accepted. Make sure that the submission is right on the first or second try. One big source of errors is not changing the template name of the repository for a link to your repository.

We can see that some users that created 14 issues didn’t have any package approved.

Around 50% of the ~400 yearly packages submitted are approved.

So if you have a package that fits Bioconductor you’ll get fairly well on the submission process. But I always recommend the lengthy pages for developers about the submission process and the packages guidelinines and requirements. Also make sure that you understand the expectations on what to fill of the template and what to expect after submitting: Don’t post several issues for the same package if you haven’t got any reply, but you can send a reminder on the issue and an email to the mailing list. Many times the reason for not accepting a package is because they do not follow the guidelines.

Reviewers

If your package fits Bioconductor a bot will assign a reviewer to check the package. Many submissions (~33%) do not get a reviewer assigned and the package is not included on Bioconductor’s repository, most (~90%) of the cases this happens the same day of the submission. But once they get one reviewer ~82% of them are approved.

There core members of Bioconductor are 9 usual reviewers:

Probably you might get assigned the project leader Martin Morgan, but you can work with any other reviewer, as the work load is quite distributed: mtmorgan, lshep, LiNk-NY, nturaga, dvantwisk, hpages, Kayla-Morrell, Liubuntu, vobencha.

The reviews are fairly done by all the members. We can see that all of them approves more than 75% of the packages assigned.

Most of the reviewers take few days until the issue is closed.

Looking at these plots more time reviewing do not mean closing the issue without being accepted.

Correction! If we divide by approved submissions, some reviewers close later submissions that are not approved. They might wait longer to close the issues without approving them, or maybe they prefer to suggest modifications and a second submission of the same package. Others close faster than they accept the packages but as you can see by the error bars it is quite variable. In general you can expect more than 40 days review.

They might accept faster packages but it is because they provide faster feedback with more comments?

Many of the comments I suspect are automatic messages from the bot about building and receiving commits. But apparently there isn’t a difference between them.

Here almost all the reviewers have more comments on approved packages. Also, there is less variability on the number of comments than on the time they are open. Answering doubts and giving feedback to the reviewers and contributors increases acceptance, even if this comment increase is due to automatic messages.

Usually more than 20 comments on the issue will be enough but the more feedback, the better.

Taking both information together we can see the pattern for all the reviewers:

Most of the not approved issues have less comments and usually remain less days open.

Rushing?

Reviewers take their time to accept or reject the packages. But do contributors have some rush to get them accepted at the end of an academic year, before a Bioconductor release or before a publication? While I cannot check if there is an article accompanying the package, I can check other time trends.

So most days there is a new package submission and on the best day there are 8. There doesn’t seem to be a seasonality towards the end of the course, maybe because the contributors are international and do not share the same schedule or holidays.

Using the dates of new Bioconductor releases we can check if there is some rush to submit packages closer to the new release:

The vertical line indicates the usual time the submissions are no longer accepted (around 30 days before the release day). So it seems like people submit close to the date but much consistently around the year as previously seen.

Those that submit closer to the date of release have higher rates of not being included on Bioconductor.

Very few packages went to the worst scenario: submitted right before a deadline and then it wasn’t accepted until many releases later. However there is a curious effect, the closer the submission is to the release day, the shorter are the reviews of the accepted packages.

Here we can observe this effect on the latest release.

I expected to see longer review time for the issues submitted closer to the release but we don’t see it. This indicates a sustained effort to review the submissions.

Pitfalls

We have seen mostly the path to success, but in order to improve the process we must look what lead to having the packages not approved.

We can check if packages

When the user that submits the package is the owner of the repository then it is more likely that the package is accepted. Probably because it is easier to manage the version control system that way.

Most packages are closed fast but some review take longer than a year! Those closed fast are automatically done by the bot due to several reasons. Among these reasons:

Table 1: Issues with more than one package
Number of packages	Times
0	13
2	11

Table 2: Multiple submission for the same package
Name	Packages
yourpackagename	68
cytofkit2	8
test-package	7
PoTRA	6
PFP	6
Summix	6

It is equally probable to not provide a link or to provide two. However many times the template is used as is and the link to the name of the repository remains “yourpackagename”, which is detected by the bot and closed.

Table 3: Comments on not approved submissions
Comments	Submissions
1	413
2	42
3	26
4	13
5	11
6	9
12	9
10	8
0	7
7	7

Most of the rejected submissions is done automatically by the “bioc-issue-bot”, with a brief message. But in general few messages are written before closing the submissions.

Table 4: Ending of rejected issues
Ending	Number of issues
.git	20
.R	4
.data	3
.Gmax.EnsemblPlants.Gmv2	3
.data.oma.All.Jan2020	2
.2	1

Sometimes the user provides a link to a compressed file, or it ends incorrectly with .git as when the url need to clone a repository.

Table 5: Labels of rejected packages
Number of labels	Number of issues
0	455
1	81
2	75
3	41
4	8
5	2

The bot automatically labels the issues to provide information about the build and check of the package and about the stage of the project. Usually this means they do not get assigned a reviewer or built on the Bioconductor server.

Looking at which labels go together, if the review starts the next halting point is an error on the build process, probably the issue is not fixed and the submission not approved.

Conclusion

Most of the packages submitted to Bioconductor are accepted. Those that are not accepted are usually due to submission formatting issues detected automatically by the bioc-issue-bot. Reviewers provide a lot of feedback that if followed leads to acceptance of the package, when this feedback is ignored it leads to a rejection of the submission. Some packages are submitted right before a release and miss it, while others undergo a long review and miss several releases, but if you want to have your package included on the next release, aim to submit 40 days before the release date (usually on the end of April and October).

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.1 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2021-01-08                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date       lib source                           
##  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.1)                   
##  backports     1.2.1   2020-12-09 [1] CRAN (R 4.0.1)                   
##  beeswarm      0.2.3   2016-04-25 [1] CRAN (R 4.0.1)                   
##  BiocManager * 1.30.10 2019-11-16 [1] CRAN (R 4.0.1)                   
##  blogdown      0.21.84 2021-01-07 [1] Github (rstudio/blogdown@c4fbb58)
##  bookdown      0.21    2020-10-13 [1] CRAN (R 4.0.1)                   
##  broom         0.7.3   2020-12-16 [1] CRAN (R 4.0.1)                   
##  cellranger    1.1.0   2016-07-27 [1] CRAN (R 4.0.1)                   
##  cli           2.2.0   2020-11-20 [1] CRAN (R 4.0.1)                   
##  codetools     0.2-18  2020-11-04 [1] CRAN (R 4.0.1)                   
##  colorspace    2.0-0   2020-11-11 [1] CRAN (R 4.0.1)                   
##  crayon        1.3.4   2017-09-16 [1] CRAN (R 4.0.1)                   
##  curl          4.3     2019-12-02 [1] CRAN (R 4.0.1)                   
##  DBI           1.1.0   2019-12-15 [1] CRAN (R 4.0.1)                   
##  dbplyr        2.0.0   2020-11-03 [1] CRAN (R 4.0.1)                   
##  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.1)                   
##  dplyr       * 1.0.2   2020-08-18 [1] CRAN (R 4.0.1)                   
##  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.1)                   
##  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.1)                   
##  fansi         0.4.1   2020-01-08 [1] CRAN (R 4.0.1)                   
##  farver        2.0.3   2020-01-16 [1] CRAN (R 4.0.1)                   
##  forcats     * 0.5.0   2020-03-01 [1] CRAN (R 4.0.1)                   
##  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.1)                   
##  generics      0.1.0   2020-10-31 [1] CRAN (R 4.0.1)                   
##  ggbeeswarm    0.6.0   2017-08-07 [1] CRAN (R 4.0.1)                   
##  ggplot2     * 3.3.2   2020-06-19 [1] CRAN (R 4.0.1)                   
##  gh          * 1.2.0   2020-11-27 [1] CRAN (R 4.0.1)                   
##  gitcreds      0.1.1   2020-12-04 [1] CRAN (R 4.0.1)                   
##  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  gridExtra     2.3     2017-09-09 [1] CRAN (R 4.0.1)                   
##  gtable        0.3.0   2019-03-25 [1] CRAN (R 4.0.1)                   
##  haven         2.3.1   2020-06-01 [1] CRAN (R 4.0.1)                   
##  highr         0.8     2019-03-20 [1] CRAN (R 4.0.1)                   
##  hms           0.5.3   2020-01-08 [1] CRAN (R 4.0.1)                   
##  htmltools     0.5.0   2020-06-16 [1] CRAN (R 4.0.1)                   
##  httr          1.4.2   2020-07-20 [1] CRAN (R 4.0.1)                   
##  jsonlite      1.7.2   2020-12-09 [1] CRAN (R 4.0.1)                   
##  knitr         1.30    2020-09-22 [1] CRAN (R 4.0.1)                   
##  labeling      0.4.2   2020-10-20 [1] CRAN (R 4.0.1)                   
##  lifecycle     0.2.0   2020-03-06 [1] CRAN (R 4.0.1)                   
##  lubridate   * 1.7.9.2 2020-11-13 [1] CRAN (R 4.0.1)                   
##  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.1)                   
##  modelr        0.1.8   2020-05-19 [1] CRAN (R 4.0.1)                   
##  munsell       0.5.0   2018-06-12 [1] CRAN (R 4.0.1)                   
##  pillar        1.4.7   2020-11-20 [1] CRAN (R 4.0.1)                   
##  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.1)                   
##  plyr          1.8.6   2020-03-03 [1] CRAN (R 4.0.1)                   
##  purrr       * 0.3.4   2020-04-17 [1] CRAN (R 4.0.1)                   
##  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.1)                   
##  Rcpp          1.0.5   2020-07-06 [1] CRAN (R 4.0.1)                   
##  readr       * 1.4.0   2020-10-05 [1] CRAN (R 4.0.1)                   
##  readxl        1.3.1   2019-03-13 [1] CRAN (R 4.0.1)                   
##  reprex        0.3.0   2019-05-16 [1] CRAN (R 4.0.1)                   
##  rlang         0.4.10  2020-12-30 [1] CRAN (R 4.0.1)                   
##  rmarkdown     2.6     2020-12-14 [1] CRAN (R 4.0.1)                   
##  rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.0.1)                   
##  rvest         0.3.6   2020-07-25 [1] CRAN (R 4.0.1)                   
##  scales        1.1.1   2020-05-11 [1] CRAN (R 4.0.1)                   
##  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.1)                   
##  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.1)                   
##  stringr     * 1.4.0   2019-02-10 [1] CRAN (R 4.0.1)                   
##  tibble      * 3.0.4   2020-10-12 [1] CRAN (R 4.0.1)                   
##  tidyr       * 1.1.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  tidyselect    1.1.0   2020-05-11 [1] CRAN (R 4.0.1)                   
##  tidyverse   * 1.3.0   2019-11-21 [1] CRAN (R 4.0.1)                   
##  UpSetR      * 1.4.0   2019-05-22 [1] CRAN (R 4.0.1)                   
##  utf8          1.1.4   2018-05-24 [1] CRAN (R 4.0.1)                   
##  vctrs         0.3.6   2020-12-17 [1] CRAN (R 4.0.1)                   
##  vipor         0.4.5   2017-03-22 [1] CRAN (R 4.0.1)                   
##  withr         2.3.0   2020-09-22 [1] CRAN (R 4.0.1)                   
##  xfun          0.20    2021-01-06 [1] CRAN (R 4.0.1)                   
##  xml2          1.3.2   2020-04-23 [1] CRAN (R 4.0.1)                   
##  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.1)                   
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library