Bioconductor | B101nfo

Bioconductor submissions: reviews

Fri, 31 Jul 2020 00:00:00 +0000

First post is on Bioconductor submissions raised some questions comments but at the time I didn’t have a good way to answer them:

issues get closed after they got assigned a reviewer and before the reviewer actually gets a chance to start the review.
issues assigned to multiple people or issues that switched reviewers

To answer both of them I needed more information. On the previous post I only gathered the information of the state of the issues at that moment. This excluded label changes, reviewer changes, renaming the issues, who commented on the issues and many more things. To retrieve these information from github I developed a new package {socialGH} which downloads it from Github to make this analysis possible.

I gathered all the information on issues using {socialGH} on the 2020/08/18 and stored them in a tidy format. Now that we have more data about the issues we can plot them similarly to what we did on the previous post:

We can see that sometimes the issues remained silent for several months and then had a high level of events, or a single one (closing mainly).

Compare to the previous version it is surprising to see that one of the earliest issues still gets new events to date. Apparently issue 51, and others, are being used to test the Bioconductor builder or the bot used to automate the process.

We can see that some reviewers comment more and are involved in more issues, but there is a group of fairly similar comments and issues.

Clearly this will not be the same with users who submit packages:

We can clearly see that most users are mainly only involved on submitting packages on one issue, but some people participate a lot and/or in several issues.

Events

We cab count how many users and issues have been involved with each event type:

We can see that few issues are locked (remember that it is different than closing the issue) or comments deleted. There is an almost equal amount of labeling and unlabeling, performed by few people but on lots of issues. There is a group of mentioning, commenting and subscribing too. Many people is involved on commenting and creating submissions.

Looking a bit further on the issues we can look at the events that take place:

We can see that most issues have few events which agrees with the previous findings that issues are handled fairly and expeditiously.

The most common event are comments, mentions and the rarest events are deleting comments, locking or referencing them.

We can see that most issues get around 50 events, but we can clearly see lots of issues that receive very few events.

However, looking at the distribution along time most of the events are on the first 6 months after opening the issue.

Some submissions have many events while others have few events but last longer. This is without looking at issues that were closed and later reopened, or the involvement of bioc-issue-bot. We can see that in a short amount of days a lot of events can be triggered. This is mostly that for each version update there are at least two messages on the issue.

Reviews

Most of the differences are due to the automatic or manual review of the submissions.

After passing some preliminary checks a reviewer is assigned to manually review the package:

We can see that usually some adjustment to reviewers is made usually changing them but some times they are added to have more than 1 reviewer (194). Almost all the submitted packages without a reviewer (537 submissions) were rejected except for three: 81, 82, 83 that were reviewed by vobencha even if not officially assigned. Some of these were rejected because they didn’t pass some automatic check and other after preliminary inspection.

If some of these preliminary checks aren’t corrected the issue is closed:

We can see that is common to close and reopen issues, but they are generally done by the same people (the reviewer, the original submitter or the bot).

On submissions with a reviewer assigned that were closed by someone (not the submitter), we can see here that reviews with more than one reviewer do not lead to significantly more comments from the reviewers.

Although it seems like this leads to have the package approved.

We can see that there are slightly more comments from reviewers on the approved packages. Perhaps because they provide more feedback once the review starts.

We can see that the percentage of approved on those issues where the reviewer commented is fairly similar and around 80% or above. However there is also at least a 5% of submissions that are closed despite the comments from reviewers. Usually this has to be with unresponsiveness from the submitter:

We can see that on the approved packages there are usually more events on the same time period. So make sure to follow the advice of the reviewers and the bot and make all the errors and warnings disappear.

While most issues have at least 8 different events they usually have 4 users involved. Presumably the creator of the issue, bioc-issue-bot, a reviewer and someone else.

The more users involved, more different type of events are triggered. Presumably more people get subscribed or is mentioned.

As expected the more users are involved in an issue more events are produced. We can also see the submissions that are closed by the bioc-issue-bot with few comments and users.

Who does each action ?

We can look now at who performs what, we know there are 951 participants:

Here I cut the top 35 people who have triggered more than events. We can clearly see who are official reviewers (as seen on the previous post) because they labeled and unlabeled issues. bioc-issue-bot is an special user that makes lots of comments and assigning reviewers to issues. We can also see that it previously renamed the issues or unassigned reviewers.

Comments by users (excluding the bot), usually only the author, the reviewer comment, frequently mtmorgan also suggests.

We can see that Martin Morgan has commented on almost all the submissions. The odd issue where there are some comments from mtmorgan but no other is the issue 611 where he is the reviewer and the submitter of a package.

Bioconductor Bot

We have seen that one of the most relevant “users” is bioc-issue-bot.
It is a bot that performs and report automatic checks on the submissions (The code can be found here). Let’s explore what does and what does it report

Classifying the comments reports that most of the comments are build results or that received a valid push. We can also see some changes on the bot, like changing the messages or reporting differently the process triggered. However, it also reports common problems on the submission: missing repository, unmatch between the repository name and the package name, reposting the same package, missing SSH key (needed to be able to push to Bioconductor git server), … We can see that most of them are related to building the packages being submitted.

Let’s check if there is some differences on the comments according to if they are later accepted or not.

We can see clearly differences on the behavior of the issues, some type of comments are all of the non-accepted packages. It seems like some are not corrected. We can see here some common areas when submissions fail: failing to provide a valid link to the repository or many links to several repositories, or failing to comply with the guidelines about the repository name and the package name. Reposting the same package is also a common reason of not getting that submission approved.

The only significant are the build results, the build results, the valid push, received, unmatch between the package name and the repository, and the update version.

We can see, (and I expected so) that comments of bioc-issue-bot are driven by the build system.

But the other comments are more frequent among the not approved packages, mainly errors and other automatically detected problems.

Aside from the bot, other users might comment on issues too:

In some issues reviewers comment more than authors, while on some there are more comments from authors than reviewers. Surprisingly, in some issues there are more comments from other users than authors or reviewers.

Usually packages not approved have less comments from authors, reviewers and other users.

About the time to get feedback we have previously seen that most of the issues were fast. However it included automatic actions from the bot, so let’s check again without the automatic events.

Here the squares indicate when a issue is closed, and we can see most comments are before the issues are closed.

And there doesn’t seem to be any difference between approved and not approved packages.

I would have expected more comments after closing an issue in the not approved submissions. This might indicate that the discussion happens before and/or that the process after closing the issue is not clear enough.

Steps

So far we have compared closed and open issues, users, comments, events, but we haven’t looked how does the proccess goes.

Let’s see how much time does it take to start the review and the time till the reviewer comments:

Successful build?	Approved?	Submissions
No	No	608
Yes	No	78
Yes	Ongoing	28
No	Ongoing	19
Yes	Yes	723
No	Yes	125

We can see that most not approved packages do not have a successful build.

An overview of the submissions, when was the first successful build and when was the first comment of the reviewer after the successful build.

Usually it takes close to 3 days to have the first successful build on Bioconductor (if there is any as we have seen).

After it it takes close to 10 days to the reviewer to comment.

This comment might be the review of the package, as it is only done after the submissions passes all the checks.

A we have seen on the first table the main blocking point is that many submitted packages do not successfully build on Bioconductor servers.
But as we can see approved packages built slightly earlier. However the time between the first successful built and the first reviewer comment might be different, let’s check:

We can see there aren’t many differences on the time between the first successful built and the first reviewer comment. Once a package has built and has the review it takes some time to address the comments till it is accepted:

We can see that since submission it usually takes two weeks to have the package accepted. However, the time between the first reviewer comment (after the first successful built) and the acceptance of the package might be different:

We can see that few packages take more than 50 days till acceptance. To have a complete picture we can see all the issues on a single plot with how much do they take to move to the next phase:

We can see odd packages like the one that built correctly in short time but had some issues and finally got renamed and submitted on a different issue, or the issue that took more than 100 days to build correctly for the first time (and was later rejected).

We can see that the steps that take more time is doing the review and later modify the package to address the comments pointed by the reviewers.

Approved	Succcessful built (days)	First reviewer comment after build (days)	Accepted (days)	Time between build and reviewer comment (days)	Time between review and acceptance (days)
No	5	19	NA	14	NA
Ongoing	3	25	NA	19	NA
Yes	3	14	37	10	19

Looking at the median times of each step we can clearly see the same pattern.

Labels

So far we have looked via a combination of comments from reviewers and labels. However, the official way of showing on which step is a review is using labels.

We have seen is the second most frequent event, around 1100 submissions have at least one label annotation. We can see here how many:

We can see that the version bump, and the state of the build are the most frequent labels.

Looking into each label we can see differences between acceoted and declined submissions:

Here we can see the differences in label assignment on each issue (labels which are assigned only to approved or rejected packages are not shown).

Some submissions were initially declined but later got approved, as well as some issues that for some time went inactive but ended up being accepted.

The main difference is how many times there are errors, timeouts, warnings, version bump required or OK on the issues approved. Successful packages have more! If you submit a package and get those errors don’t worry, it is normal!!

Approved	VERSION BUMP REQUIRED	ERROR	WARNINGS	OK	TIMEOUT	ABNORMAL	1. awaiting moderation	3a. accepted	2. review in progress	3c. inactive	3b. declined
No	0	1	0	0	0	0	1	0	1	0	0
Yes	2	2	1	2	0	0	1	1	1	0	0

We can see that is more common to have more “troubles” on packages that end up accepted than those than not. Note that there are at least two OK per submission, one required before the review start and the other one after the changes before acceptance.

Conclusions

Most of the problems with the submissions are formal and automatically detected by the bot. Next come problems from the package itself not passing the checks performed on Bioconductor. So if you want to have your package included make sure that the package builds on Bioconductor and respond fast to the feedback provided by the bot. Once your package successfully builds, then address the comments from the reviewer. Follow the steps and if you don’t drop out you’ll see your package accepted.

On the next post I’ll explore the data from rOpenSci, which also does the reviews on GitHub.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.1 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2021-01-08                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package      * version date       lib source                           
##  assertthat     0.2.1   2019-03-21 [1] CRAN (R 4.0.1)                   
##  backports      1.2.1   2020-12-09 [1] CRAN (R 4.0.1)                   
##  blogdown       0.21.84 2021-01-07 [1] Github (rstudio/blogdown@c4fbb58)
##  bookdown       0.21    2020-10-13 [1] CRAN (R 4.0.1)                   
##  broom          0.7.3   2020-12-16 [1] CRAN (R 4.0.1)                   
##  cellranger     1.1.0   2016-07-27 [1] CRAN (R 4.0.1)                   
##  cli            2.2.0   2020-11-20 [1] CRAN (R 4.0.1)                   
##  codetools      0.2-18  2020-11-04 [1] CRAN (R 4.0.1)                   
##  colorspace     2.0-0   2020-11-11 [1] CRAN (R 4.0.1)                   
##  crayon         1.3.4   2017-09-16 [1] CRAN (R 4.0.1)                   
##  DBI            1.1.0   2019-12-15 [1] CRAN (R 4.0.1)                   
##  dbplyr         2.0.0   2020-11-03 [1] CRAN (R 4.0.1)                   
##  digest         0.6.27  2020-10-24 [1] CRAN (R 4.0.1)                   
##  dplyr        * 1.0.2   2020-08-18 [1] CRAN (R 4.0.1)                   
##  ellipsis       0.3.1   2020-05-15 [1] CRAN (R 4.0.1)                   
##  evaluate       0.14    2019-05-28 [1] CRAN (R 4.0.1)                   
##  fansi          0.4.1   2020-01-08 [1] CRAN (R 4.0.1)                   
##  farver         2.0.3   2020-01-16 [1] CRAN (R 4.0.1)                   
##  forcats      * 0.5.0   2020-03-01 [1] CRAN (R 4.0.1)                   
##  fs             1.5.0   2020-07-31 [1] CRAN (R 4.0.1)                   
##  generics       0.1.0   2020-10-31 [1] CRAN (R 4.0.1)                   
##  ggplot2      * 3.3.3   2020-12-30 [1] CRAN (R 4.0.1)                   
##  ggrepel      * 0.9.0   2020-12-16 [1] CRAN (R 4.0.1)                   
##  gh             1.2.0   2020-11-27 [1] CRAN (R 4.0.1)                   
##  glue           1.4.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  gtable         0.3.0   2019-03-25 [1] CRAN (R 4.0.1)                   
##  haven          2.3.1   2020-06-01 [1] CRAN (R 4.0.1)                   
##  here           1.0.1   2020-12-13 [1] CRAN (R 4.0.1)                   
##  highr          0.8     2019-03-20 [1] CRAN (R 4.0.1)                   
##  hms            0.5.3   2020-01-08 [1] CRAN (R 4.0.1)                   
##  htmltools      0.5.0   2020-06-16 [1] CRAN (R 4.0.1)                   
##  httr           1.4.2   2020-07-20 [1] CRAN (R 4.0.1)                   
##  jsonlite       1.7.2   2020-12-09 [1] CRAN (R 4.0.1)                   
##  knitr          1.30    2020-09-22 [1] CRAN (R 4.0.1)                   
##  labeling       0.4.2   2020-10-20 [1] CRAN (R 4.0.1)                   
##  lifecycle      0.2.0   2020-03-06 [1] CRAN (R 4.0.1)                   
##  lubridate      1.7.9.2 2020-11-13 [1] CRAN (R 4.0.1)                   
##  magrittr       2.0.1   2020-11-17 [1] CRAN (R 4.0.1)                   
##  modelr         0.1.8   2020-05-19 [1] CRAN (R 4.0.1)                   
##  munsell        0.5.0   2018-06-12 [1] CRAN (R 4.0.1)                   
##  patchwork    * 1.1.1   2020-12-17 [1] CRAN (R 4.0.1)                   
##  pillar         1.4.7   2020-11-20 [1] CRAN (R 4.0.1)                   
##  pkgconfig      2.0.3   2019-09-22 [1] CRAN (R 4.0.1)                   
##  purrr        * 0.3.4   2020-04-17 [1] CRAN (R 4.0.1)                   
##  R6             2.5.0   2020-10-28 [1] CRAN (R 4.0.1)                   
##  RColorBrewer   1.1-2   2014-12-07 [1] CRAN (R 4.0.1)                   
##  Rcpp           1.0.5   2020-07-06 [1] CRAN (R 4.0.1)                   
##  readr        * 1.4.0   2020-10-05 [1] CRAN (R 4.0.1)                   
##  readxl         1.3.1   2019-03-13 [1] CRAN (R 4.0.1)                   
##  reprex         0.3.0   2019-05-16 [1] CRAN (R 4.0.1)                   
##  rlang          0.4.10  2020-12-30 [1] CRAN (R 4.0.1)                   
##  rmarkdown      2.6     2020-12-14 [1] CRAN (R 4.0.1)                   
##  rprojroot      2.0.2   2020-11-15 [1] CRAN (R 4.0.1)                   
##  rstudioapi     0.13    2020-11-12 [1] CRAN (R 4.0.1)                   
##  rvest          0.3.6   2020-07-25 [1] CRAN (R 4.0.1)                   
##  scales         1.1.1   2020-05-11 [1] CRAN (R 4.0.1)                   
##  sessioninfo    1.1.1   2018-11-05 [1] CRAN (R 4.0.1)                   
##  socialGH     * 0.0.3   2020-08-17 [1] local                            
##  stringi        1.5.3   2020-09-09 [1] CRAN (R 4.0.1)                   
##  stringr      * 1.4.0   2019-02-10 [1] CRAN (R 4.0.1)                   
##  tibble       * 3.0.4   2020-10-12 [1] CRAN (R 4.0.1)                   
##  tidyr        * 1.1.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  tidyselect     1.1.0   2020-05-11 [1] CRAN (R 4.0.1)                   
##  tidyverse    * 1.3.0   2019-11-21 [1] CRAN (R 4.0.1)                   
##  vctrs          0.3.6   2020-12-17 [1] CRAN (R 4.0.1)                   
##  viridisLite    0.3.0   2018-02-01 [1] CRAN (R 4.0.1)                   
##  withr          2.3.0   2020-09-22 [1] CRAN (R 4.0.1)                   
##  xfun           0.20    2021-01-06 [1] CRAN (R 4.0.1)                   
##  xml2           1.3.2   2020-04-23 [1] CRAN (R 4.0.1)                   
##  yaml           2.2.1   2020-02-01 [1] CRAN (R 4.0.1)                   
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library

Bioconductor submissions

Mon, 15 Jun 2020 00:00:00 +0000

The other day I was curious about how many submissions are on Bioconductor and how do they work, is there any pattern toward the release cycle or not? In this brief post I’ll explore some data available from the github repository where the submissions are done as issues. I mostly focus on what happens when one submits a package so that it can help future submissions.

There are around 1826 issues, I extract the interesting information: date of creation, who creates them, where is the linked package. when are they closed, labels at the moment, reviewers…

We can further mark those approved because they are on bioconductor but do not have the approved label (Sometimes they get accepted but they forget to change labels on the issue).

I didn’t mark as accepted packages that were submitted several times (184 packages have been submitted more than once). Some of them are due to being on the old tracker and were not approved on this repository.
Using also the CRAN repository we could find which packages where submitted to Bioconductor but end up on CRAN. But as I expect a low number of these I won’t check them.

Exploring the submissions

First of all a general overview of the data available:

On this plot we can see almost everything, date of creation, time opened, when was closed, in which release is included if accepted and the rate of submissions to Bioconductor. What we miss is about the authors submitting the packages and about the packages themselves.

Submitting authors

One of the core strength of R is the open community. There have been 749 different users submitting at least one package.

We can clearly see that most authors submit just one package but some submit a few more, with an outlier who have submitted more than 10 packages.

Most of the authors get their submission included at Bioconductor at the first try. Few need two submissions and some do not get their package included on Bioconductor. This can be because they withdraw the submission, or they decide to submit elsewhere or simply they do not go through the review.

If authors submit more packages they usually get them approved. Although we can also see some users that make several submissions.

Regardless if the package is submitted just once or several times. More submissions do not make the package get accepted. Make sure that the submission is right on the first or second try. One big source of errors is not changing the template name of the repository for a link to your repository.

We can see that some users that created 14 issues didn’t have any package approved.

Around 50% of the ~400 yearly packages submitted are approved.

So if you have a package that fits Bioconductor you’ll get fairly well on the submission process. But I always recommend the lengthy pages for developers about the submission process and the packages guidelinines and requirements. Also make sure that you understand the expectations on what to fill of the template and what to expect after submitting: Don’t post several issues for the same package if you haven’t got any reply, but you can send a reminder on the issue and an email to the mailing list. Many times the reason for not accepting a package is because they do not follow the guidelines.

Reviewers

If your package fits Bioconductor a bot will assign a reviewer to check the package. Many submissions (~33%) do not get a reviewer assigned and the package is not included on Bioconductor’s repository, most (~90%) of the cases this happens the same day of the submission. But once they get one reviewer ~82% of them are approved.

There core members of Bioconductor are 9 usual reviewers:

Probably you might get assigned the project leader Martin Morgan, but you can work with any other reviewer, as the work load is quite distributed: mtmorgan, lshep, LiNk-NY, nturaga, dvantwisk, hpages, Kayla-Morrell, Liubuntu, vobencha.

The reviews are fairly done by all the members. We can see that all of them approves more than 75% of the packages assigned.

Most of the reviewers take few days until the issue is closed.

Looking at these plots more time reviewing do not mean closing the issue without being accepted.

Correction! If we divide by approved submissions, some reviewers close later submissions that are not approved. They might wait longer to close the issues without approving them, or maybe they prefer to suggest modifications and a second submission of the same package. Others close faster than they accept the packages but as you can see by the error bars it is quite variable. In general you can expect more than 40 days review.

They might accept faster packages but it is because they provide faster feedback with more comments?

Many of the comments I suspect are automatic messages from the bot about building and receiving commits. But apparently there isn’t a difference between them.

Here almost all the reviewers have more comments on approved packages. Also, there is less variability on the number of comments than on the time they are open. Answering doubts and giving feedback to the reviewers and contributors increases acceptance, even if this comment increase is due to automatic messages.

Usually more than 20 comments on the issue will be enough but the more feedback, the better.

Taking both information together we can see the pattern for all the reviewers:

Most of the not approved issues have less comments and usually remain less days open.

Rushing?

Reviewers take their time to accept or reject the packages. But do contributors have some rush to get them accepted at the end of an academic year, before a Bioconductor release or before a publication? While I cannot check if there is an article accompanying the package, I can check other time trends.

So most days there is a new package submission and on the best day there are 8. There doesn’t seem to be a seasonality towards the end of the course, maybe because the contributors are international and do not share the same schedule or holidays.

Using the dates of new Bioconductor releases we can check if there is some rush to submit packages closer to the new release:

The vertical line indicates the usual time the submissions are no longer accepted (around 30 days before the release day). So it seems like people submit close to the date but much consistently around the year as previously seen.

Those that submit closer to the date of release have higher rates of not being included on Bioconductor.

Very few packages went to the worst scenario: submitted right before a deadline and then it wasn’t accepted until many releases later. However there is a curious effect, the closer the submission is to the release day, the shorter are the reviews of the accepted packages.

Here we can observe this effect on the latest release.

I expected to see longer review time for the issues submitted closer to the release but we don’t see it. This indicates a sustained effort to review the submissions.

Pitfalls

We have seen mostly the path to success, but in order to improve the process we must look what lead to having the packages not approved.

We can check if packages

When the user that submits the package is the owner of the repository then it is more likely that the package is accepted. Probably because it is easier to manage the version control system that way.

Most packages are closed fast but some review take longer than a year! Those closed fast are automatically done by the bot due to several reasons. Among these reasons:

Table 1: Issues with more than one package
Number of packages	Times
0	13
2	11

Table 2: Multiple submission for the same package
Name	Packages
yourpackagename	68
cytofkit2	8
test-package	7
PoTRA	6
PFP	6
Summix	6

It is equally probable to not provide a link or to provide two. However many times the template is used as is and the link to the name of the repository remains “yourpackagename”, which is detected by the bot and closed.

Table 3: Comments on not approved submissions
Comments	Submissions
1	413
2	42
3	26
4	13
5	11
6	9
12	9
10	8
0	7
7	7

Most of the rejected submissions is done automatically by the “bioc-issue-bot”, with a brief message. But in general few messages are written before closing the submissions.

Table 4: Ending of rejected issues
Ending	Number of issues
.git	20
.R	4
.data	3
.Gmax.EnsemblPlants.Gmv2	3
.data.oma.All.Jan2020	2
.2	1

Sometimes the user provides a link to a compressed file, or it ends incorrectly with .git as when the url need to clone a repository.

Table 5: Labels of rejected packages
Number of labels	Number of issues
0	455
1	81
2	75
3	41
4	8
5	2

The bot automatically labels the issues to provide information about the build and check of the package and about the stage of the project. Usually this means they do not get assigned a reviewer or built on the Bioconductor server.

Looking at which labels go together, if the review starts the next halting point is an error on the build process, probably the issue is not fixed and the submission not approved.

Conclusion

Most of the packages submitted to Bioconductor are accepted. Those that are not accepted are usually due to submission formatting issues detected automatically by the bioc-issue-bot. Reviewers provide a lot of feedback that if followed leads to acceptance of the package, when this feedback is ignored it leads to a rejection of the submission. Some packages are submitted right before a release and miss it, while others undergo a long review and miss several releases, but if you want to have your package included on the next release, aim to submit 40 days before the release date (usually on the end of April and October).

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.1 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2021-01-08                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date       lib source                           
##  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.1)                   
##  backports     1.2.1   2020-12-09 [1] CRAN (R 4.0.1)                   
##  beeswarm      0.2.3   2016-04-25 [1] CRAN (R 4.0.1)                   
##  BiocManager * 1.30.10 2019-11-16 [1] CRAN (R 4.0.1)                   
##  blogdown      0.21.84 2021-01-07 [1] Github (rstudio/blogdown@c4fbb58)
##  bookdown      0.21    2020-10-13 [1] CRAN (R 4.0.1)                   
##  broom         0.7.3   2020-12-16 [1] CRAN (R 4.0.1)                   
##  cellranger    1.1.0   2016-07-27 [1] CRAN (R 4.0.1)                   
##  cli           2.2.0   2020-11-20 [1] CRAN (R 4.0.1)                   
##  codetools     0.2-18  2020-11-04 [1] CRAN (R 4.0.1)                   
##  colorspace    2.0-0   2020-11-11 [1] CRAN (R 4.0.1)                   
##  crayon        1.3.4   2017-09-16 [1] CRAN (R 4.0.1)                   
##  curl          4.3     2019-12-02 [1] CRAN (R 4.0.1)                   
##  DBI           1.1.0   2019-12-15 [1] CRAN (R 4.0.1)                   
##  dbplyr        2.0.0   2020-11-03 [1] CRAN (R 4.0.1)                   
##  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.1)                   
##  dplyr       * 1.0.2   2020-08-18 [1] CRAN (R 4.0.1)                   
##  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.1)                   
##  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.1)                   
##  fansi         0.4.1   2020-01-08 [1] CRAN (R 4.0.1)                   
##  farver        2.0.3   2020-01-16 [1] CRAN (R 4.0.1)                   
##  forcats     * 0.5.0   2020-03-01 [1] CRAN (R 4.0.1)                   
##  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.1)                   
##  generics      0.1.0   2020-10-31 [1] CRAN (R 4.0.1)                   
##  ggbeeswarm    0.6.0   2017-08-07 [1] CRAN (R 4.0.1)                   
##  ggplot2     * 3.3.2   2020-06-19 [1] CRAN (R 4.0.1)                   
##  gh          * 1.2.0   2020-11-27 [1] CRAN (R 4.0.1)                   
##  gitcreds      0.1.1   2020-12-04 [1] CRAN (R 4.0.1)                   
##  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  gridExtra     2.3     2017-09-09 [1] CRAN (R 4.0.1)                   
##  gtable        0.3.0   2019-03-25 [1] CRAN (R 4.0.1)                   
##  haven         2.3.1   2020-06-01 [1] CRAN (R 4.0.1)                   
##  highr         0.8     2019-03-20 [1] CRAN (R 4.0.1)                   
##  hms           0.5.3   2020-01-08 [1] CRAN (R 4.0.1)                   
##  htmltools     0.5.0   2020-06-16 [1] CRAN (R 4.0.1)                   
##  httr          1.4.2   2020-07-20 [1] CRAN (R 4.0.1)                   
##  jsonlite      1.7.2   2020-12-09 [1] CRAN (R 4.0.1)                   
##  knitr         1.30    2020-09-22 [1] CRAN (R 4.0.1)                   
##  labeling      0.4.2   2020-10-20 [1] CRAN (R 4.0.1)                   
##  lifecycle     0.2.0   2020-03-06 [1] CRAN (R 4.0.1)                   
##  lubridate   * 1.7.9.2 2020-11-13 [1] CRAN (R 4.0.1)                   
##  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.1)                   
##  modelr        0.1.8   2020-05-19 [1] CRAN (R 4.0.1)                   
##  munsell       0.5.0   2018-06-12 [1] CRAN (R 4.0.1)                   
##  pillar        1.4.7   2020-11-20 [1] CRAN (R 4.0.1)                   
##  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.1)                   
##  plyr          1.8.6   2020-03-03 [1] CRAN (R 4.0.1)                   
##  purrr       * 0.3.4   2020-04-17 [1] CRAN (R 4.0.1)                   
##  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.1)                   
##  Rcpp          1.0.5   2020-07-06 [1] CRAN (R 4.0.1)                   
##  readr       * 1.4.0   2020-10-05 [1] CRAN (R 4.0.1)                   
##  readxl        1.3.1   2019-03-13 [1] CRAN (R 4.0.1)                   
##  reprex        0.3.0   2019-05-16 [1] CRAN (R 4.0.1)                   
##  rlang         0.4.10  2020-12-30 [1] CRAN (R 4.0.1)                   
##  rmarkdown     2.6     2020-12-14 [1] CRAN (R 4.0.1)                   
##  rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.0.1)                   
##  rvest         0.3.6   2020-07-25 [1] CRAN (R 4.0.1)                   
##  scales        1.1.1   2020-05-11 [1] CRAN (R 4.0.1)                   
##  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.1)                   
##  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.1)                   
##  stringr     * 1.4.0   2019-02-10 [1] CRAN (R 4.0.1)                   
##  tibble      * 3.0.4   2020-10-12 [1] CRAN (R 4.0.1)                   
##  tidyr       * 1.1.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  tidyselect    1.1.0   2020-05-11 [1] CRAN (R 4.0.1)                   
##  tidyverse   * 1.3.0   2019-11-21 [1] CRAN (R 4.0.1)                   
##  UpSetR      * 1.4.0   2019-05-22 [1] CRAN (R 4.0.1)                   
##  utf8          1.1.4   2018-05-24 [1] CRAN (R 4.0.1)                   
##  vctrs         0.3.6   2020-12-17 [1] CRAN (R 4.0.1)                   
##  vipor         0.4.5   2017-03-22 [1] CRAN (R 4.0.1)                   
##  withr         2.3.0   2020-09-22 [1] CRAN (R 4.0.1)                   
##  xfun          0.20    2021-01-06 [1] CRAN (R 4.0.1)                   
##  xml2          1.3.2   2020-04-23 [1] CRAN (R 4.0.1)                   
##  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.1)                   
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library