r | B101nfo

useR madrid: rtweet

Thu, 29 Feb 2024 19:00:00 +0200

This presentation was in Spanish. I shared the history of my involvement with rtweet and what is happening with the package and Twitter API.

New rtweet release: 2.0.0

Fri, 16 Feb 2024 00:00:00 +0000

This is a brief announcement of rtweet version 2.0.0. This major version changes signals the move from the API v1.1 to the API v2.

There haven’t been many changes since 1.2.1 but this is to signal that the API v1.1 is deprecated.

The previous release was a bit of a rush to meet with the requirements of CRAN maintainers to fix an error and it wasn’t polished. Some users complained that it was difficult to find what worked. In this release I focused mostly to make life easier for users:

Now there is a document the deprecated functions from API v1.1 to API v2: see help("rtweet-deprecated", "rtweet"). I also made it easier for the rtweet to work with API v2: the release of httr2 1.0.0 version helped to avoid some workarounds with the authentication process.

I also focused on updating the vignettes to the most up to date recommendations. I am not sure the streaming vignettes is up to date (but keep reading why I left it as is).

Last, following CRAN policy: if users create rtweet data they can now delete it with client_clean() and auth_clean().

Future releases

For the last year I asked the community for a co-maintainer with interest in the package. Unfortunately, people that showed some interest at the end didn’t commit to it.

At the same time I also asked for donations to support an API access. It currently costs 100€ to access most endpoints which is needed to test and develop the package. However, this is more than half of what I spend in groceries last month.
Other packages like academictwitteR are also stopping development/support. Although not archived from CRAN, it has a note in the README:

Note this repo is now ARCHVIED due to changes to the Twitter API. The paid API means open-source development of this package is no longer feasible.

Similarly without financial help and community interest I won’t invest more time on it.
This is the last version that I release. I have other interests and I would like to focus on other projects. My focus will be on updating and releasing some packages I have. I also want to focus more on my own company to help the R community (and beyond). I will write about the company shortly.

There have been some discussions on social media how to signal deprecation of packages. The only method available on CRAN that I know is to declare a package ORPHANATED. I have requested to CRAN to declared the package ORPHANATED.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       Ubuntu 22.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language en
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2024-02-24
##  pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date (UTC) lib source
##  blogdown      1.18    2023-06-19 [1] CRAN (R 4.3.1)
##  bookdown      0.37    2023-12-01 [1] CRAN (R 4.3.1)
##  bslib         0.6.1   2023-11-28 [1] CRAN (R 4.3.1)
##  cachem        1.0.8   2023-05-01 [1] CRAN (R 4.3.1)
##  cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.1)
##  digest        0.6.34  2024-01-11 [1] CRAN (R 4.3.1)
##  evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.2)
##  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.1)
##  htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.2)
##  jquerylib     0.1.4   2021-04-26 [1] CRAN (R 4.3.1)
##  jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.3.1)
##  knitr         1.45    2023-10-30 [1] CRAN (R 4.3.2)
##  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.2)
##  R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.1)
##  rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.1)
##  rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.1)
##  rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
##  sass          0.4.8   2023-12-06 [1] CRAN (R 4.3.1)
##  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.1)
##  xfun          0.42    2024-02-08 [1] CRAN (R 4.3.1)
##  yaml          2.3.8   2023-12-11 [1] CRAN (R 4.3.1)
## 
##  [1] /home/lluis/bin/R/4.3.1
##  [2] /opt/R/4.3.1/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Releasing rtweet 1.2.0

Mon, 20 Mar 2023 00:00:00 +0000

I’m very excited to announce that rtweet 1.2.0 is now available on GitHub! Install it by running:

devtools::install_github("ropensci/rtweet")

Then load it in a fresh session with:

library(rtweet)

New features

This version adds many new endpoints to retrieve data from twitter:

From lists
From tweets
About users
Also, about statistics of your own content.

You can read about them in the NEWS.

Authentication

Besides fixing a problem preventing new users to use auth_setup_default(), in this release there is a new authentication mechanism.

Some endpoints require a new authentication method not previously used by rtweet. This authentication mechanism requires setting up a client.
To support it, I have added some functions to create it, save it, and use it modelling the functions from auth_*. There is now one client provided by rtweet if you don’t want to configure your own:

client_setup_default()

Additionally, I briefly expanded the authentication vignette (vignette("auth", "rtweet")) to include a section about how to obtain the required credentials. Once you get them is pretty straight forward:

auth_oauth2 <- rtweet_oauth2(app = "my_awesome_app")

This mechanism is required by some functions which are of special interest: user_self(), tweet_bookmarked(), user_blocked(), and user_timeline().

Note that due to upstream reasons, the authentication is only valid for 2 hours. You will be asked to approve the client again after the 2 hours (and save it again!).

We can set the authentication as we usually do:

auth_as(auth_oauth2)

And start retrieving our data in Twitter!

New functions

me <- user_self()
bookmarked <- user_bookmarks(me$id, n = 120)

rtweet will make as many requests as needed and automatically paginate the results. However, if you try this you might realize that the queries are slow. These are the limits imposed by Twitter.

If you want to keep track of the progress of your query, you can use verbose = TRUE:

blocked <- user_blocked(me$id, n = Inf, verbose = TRUE)
timeline <- user_timeline(me$id, n = 800, verbose = TRUE)

It will also store the data of the requests in a temporary file, in case you lose the connection you can still recover it.

Some endpoints have a length limit on the accepted input:

bioconductor <- user_by_username("Bioconductor")
bioconductor_followers <- user_followers(bioconductor$id, n = 200)
us <- user_search(ids = bioconductor_followers$id)

Errors are, in principle, easier to understand in these new functions, thanks to the messages provided:

user_blocked(bioconductor$id, n = Inf, verbose = TRUE)

Other

These new endpoints provide access to many data which only the default information is converted to a nice table. If you request more data, via expansions and fields: replies, information about the user of a tweet, … you will have to wait next release.
You can already get them but with parse = FALSE. My intention was to provide more parsing support in this release, but I think it is better to make more releases more often.

The API also provides an endpoint to check if data stored is compliant with the Terms of Service. I started working on these endpoints after the streaming endpoints because they are important.

Side story

With the deprecation of the streaming endpoints and the function stream_tweets I implemented the first three functions using Twitter’s API v2. They use a bearer token as authentication mechanism.

Many endpoints of API v2 also use this authentication mechanism and made it easy to support them.

But there was a petition to retrieve the bookmarked tweets. That endpoint required OAuth2 mechanism and didn’t allow the use of the bearer token. It is relevant because bookmarks are not provided with the data dump you can request from Twitter. This endpoint is the only automatic way to retrieve them from Twitter if you used them!

Final

This update is the last update of rtweet. The new API plans make it impossible to continue developing and testing software like rtweet without substantial financial investment (at least USD$100/month).

More importantly, this will restrict who can use the package. I think those few users still using rtweet might also afford to pay for support or development of new features. If you are one of them you can sponsor my work in rtweet.

I will remove the package from CRAN one month after the new API enters into effect (~ 1st July). Farewell Twitter.

Accessing REDCap from R

Wed, 08 Feb 2023 00:00:00 +0000

In this post, I want to summarize some of the packages to connect to REDCap. For those who don’t know, REDCap is a database designed for clinical usage, which allows easy data collection of patients’ responses by clinicians and interactions with the patients via surveys.

It has specific features such as scheduling surveys sent to patients, compatibility with tablets and mobile phones for data entry while visiting patients, grouping data in instruments (for repeating the same questions multiple times), multiple choice and check buttons, and different arms (like paths for patients). Most importantly is relatively easy to manage by clinical administrators.

In CRAN there are ~11 packages mentioning it at the time of writing it. The purpose of this post is to help decide which packages can be helpful in which situations. This post won’t be a deep analysis or comparison of capabilities, it describes some of the best and worse features of each package.

REDCapR

REDCapR is the official package to connect to the database. It allows you to read, write and filter the requests. It has some security-related functions.

REDCapTidieR

REDCapTidieR is a package that provides summaries of tables and helps with nested tibbles data by arm. It depends on REDCapR.

tidyREDCap

tidyREDCap is a package that simplifies the tables for instruments and choose-all or choose-one question types. It is easy to make tables and it depends on REDCapR. It requires the first and last columns to make instruments.

Screenshot of a design with several instruments in a single arm (from https://www.project-redcap.org/)

REDCapExporter

REDCapExporter is a package to build a data package from a database for redistribution. It does not depend on REDCapR.

redcapAPI

redcapAPI is a package for making data accessible and analysis-ready as quickly as possible with huge documentation in a wiki but has no vignette or examples and it does not depend on REDCapR.

REDCapDM

REDCapDM is a package that provides functions to read and manage REDCap data and identify missing or extreme values as well as transform the data provided by the API. It depends on REDCapR.

ReviewR

ReviewR is a package that creates a shiny website with data from the database to explore it. It uses the REDCapR to connect to your instance.

rccola

rccola is a package to provide a secure connection to the database but it doesn’t provide any handling of the data. It uses redcapAPI to connect to the database.

Other packages

Other packages mention REDCap:

nmadb: which implements its own connection procedure for a specific REDCap database of network meta-analyses.
distcomp: Allows to do computation on a distributed data also in REDCap.
cgmanalysis: which mentions that data produced is compatible with REDCap.

Conclusion

I’m sure that many packages briefly described here can do much more than what I understood from a glance at their documentation and DESCRIPTION.

Most packages provide some data for the examples (and probably tests), while others do not. This is a technical problem that might impact users if there are no examples in the functions.

REDCapR is used by most packages to access the database, but most of the packages focus on transforming the data provided by the API (or data exported) or the exported data. It highlights that the data exported is useful but that depending on the preferences of the users it needs to be transformed for easy usage.

Los paquetes van a CRAN

Fri, 25 Nov 2022 00:00:00 +0000

The “XII Jornadas de R y I congreso de R Hispano” conference is the meeting point of useRs in Spain with many people around the country and with different background from mathematicians, engineers, forest engineering. All the talks were in Spanish, although some of the contents were in English. I submitted this abstract in Spanish:

Por lo general, compartir nuestro trabajo con la comunidad R significa enviar un paquete a un archivo (CRAN, Bioconductor u otros). CRAN es el mayor repositorio de paquetes de R que viene aceptado por defecto por R. Pero, ¿Qué hay que hacer para escribir un paquete, que CRAN lo acepte y se mantenga en CRAN?
Que un paquete se mantenga en CRAN depende de la calidad del paquete. Esto se debe a que hay que pasar un proceso de revisión. Si el buen paquete sigue las reglas y tiene una calidad de acuerdo con sus criterios, se durará. Primero, hay un chequeo inicial automático; segundo, una revisión manual más profunda del código. Luego, si las sugerencias se aplican o se responden correctamente, el paquete se incluye en el archivo.
En cada paso se utilizan algunas reglas y criterios para decidir si el paquete avanza o no. Comprender lo que dicen estas reglas, los problemas comunes y los comentarios de los revisores ayudarán a evitar enviar un paquete para que sea rechazado. Reducir la fricción entre compartir nuestro trabajo, proporcionar paquetes útiles a la comunidad y minimizar el tiempo y los esfuerzos de los revisores.
A partir de los datos históricos veremos el proceso habitual, el tiempo de espera hasta su inclusión, el número de revisiones habituales antes de ser aceptados y el porcentaje de éxito. También haremos un recorrido histórico de los paquetes de CRAN: tiempo de duración de una versión en CRAN, relación entre versiones y dependencias y el número de paquetes nuevos habituales. Para ver qué características tienen que cumplir nuestro paquete para ser incluido y que otros usuarios pueden usar nuestro código con garantías de calidad.

It was accepted as a flash presentation of just 5 minutes in a parallel session focused in programming and teaching R. The room was full and people showed their interest before and after the talk, specifically how easy would it be to keep the package in CRAN or Bioconductor. If you have other questions let me know!

Exploring CRAN's files: part 2

Thu, 28 Jul 2022 00:00:00 +0000

Introduction

In the first post of the series we briefly explored packages available on CRAN. Now I’ll focus on history of the packages and its size using the following files:

packages <- tools::CRAN_package_db()
current <- tools:::CRAN_current_db()
archive <- tools:::CRAN_archive_db()

In this part we will use two files: The current and the archive, let’s see why.

current file

The current database has has the package size, dates of modification, which I assume is date added to CRAN and user name of who last modified it. This is the same information returned by file.info

current[1, 1:10]
##     size isdir mode               mtime               ctime               atime
## A3 42810 FALSE  664 2015-08-16 23:05:54 2022-09-03 12:02:27 2022-09-03 14:00:19
##     uid  gid  uname    grname
## A3 1001 1001 hornik cranadmin

archive file

The archive database returns the same information, but as you might guess by the name it doesn’t provide information about current packages but for packages in the archive and no longer available by default.

archive[[1]]
##                     size isdir mode               mtime               ctime
## A3/A3_0.9.1.tar.gz 45252 FALSE  664 2013-02-07 10:00:29 2022-08-22 18:14:53
## A3/A3_0.9.2.tar.gz 45907 FALSE  664 2013-03-26 19:58:40 2022-08-22 18:14:53
##                                  atime  uid  gid  uname    grname
## A3/A3_0.9.1.tar.gz 2022-08-22 17:39:50 1001 1001 hornik cranadmin
## A3/A3_0.9.2.tar.gz 2022-08-22 17:39:50 1010 1001 ligges cranadmin

The date matches that available on the web’s old sources, so we can be confident of it’s meaning.

CRAN history

As we have seen there are some files about the archives of CRAN. These include information about date of modification (moving/editing) and user who did it and of course name and sometimes version of the package. These archives are the great treasure of CRAN because they help to make reproducible long time ago run experiments or analysis.

Note that I’m not totally sure that this archive contains the full record of packages, some initial packages might be missing. I’m also aware of some packages removed by CRAN which do not longer appear on this records.

Nevertheless, this should provide an accurate picture of packages available through time. Also as there is no information when a package is archived (here, there is on PACKAGES.in) so I might overestimate the packages available at any given moment.

Remember the plot about acceptance of packages on CRAN? That plot only looked at current packages available, let’s check it with all the archive:

Figure 1: Packages on CRAN archive by their addition to it. There are over 125000 archives on CRAN.

All these packages come from packages with few releases and packages with many releases. If we look at which packages had the most releases:

package	Releases
spatstat	206
Matrix	204
mgcv	162
RcppArmadillo	150
rgdal	146
nlme	143
caret	139
spdep	139
lattice	137
plotrix	131
sp	128
XML	126
Rcmdr	123
lme4	122
gstat	121
arm	119
foreign	117
party	117
maptools	113
raster	108

Surprisingly there are packages with more than 200 versions on CRAN!

Figure 2: Releases distirbution. Packages and number of releases

Most packages have 1 release, usually packages have 3, but the mean is around 6.

Given all this different versions of packages how big are all the packages on CRAN?

CRAN size

Have you ever wondered how big is CRAN? According to the memory size of the source packages all CRAN source packages are approximately 96.8 Gb.

This doesn’t include binaries for multiple architectures and OS. The package size might indicate whether the package has considerable amount of data.

Looking back to the size of the packages along time we can see this pattern:

Figure 3: Package and their median size. Archived packages have become bigger since 2014. Packages on CRAN have been getting bigger since 2017.

Packages available on CRAN are smaller than those no longer on CRAN. But versions of packages on CRAN that got archived are usually bigger than current versions. Packages no longer on CRAN are usually bigger. Median size of packages is increasing (quickly).

Figure 4: Size of package with releases. Package are usually small but seem to gain weight when updating.

Typically packages increase their size with each new release up to when they reach 50 releases. For higher releases this plot depends on very few packages and might not be representative.

Figure 5: Size of package with releases by availability. Packages no longer in CRAN are usually smaller than those in it. The continous black line is CRAN’s current threshold, while the discontinous black line is current median size.

Here we can appreciate better how packages tend to be below the CRAN threshold. There isn’t much of a difference between packages available on CRAN and those archived.

If we look at the size of package of the first release over time we’ll see a representative view:

Figure 6: Size of the first release by time. Package size increases with time with a peak around 2010 and increasing again since 2014 but still hasn’t surprased the previous record.

Package size tends to increase except for the brief period 2010-2014. Currently it increases less than before that period but is close to its maximum.

Conclusions

Most packages are not updated too much, between 1 and 3 times. But there are packages that are updated quite a lot, this might mean they are data packages and not software packages or that they have frequent minor and major updates.
Most current packages have smaller size than those archived. Packages no longer available usually had bigger size than those packages still on CRAN.
Surprisingly packages increase their size a lot till the 25 release. But also with time except for a period in 2010 and 2014. This decreasing period might be due to a change in CRAN policy.

Future parts

On future posts I’ll explore:

patterns accepting packages and updates in packages.
the relation between dependencies, initial release and updates.
who handled the packages.

Exploring CRAN's files: part 1

Sat, 23 Jul 2022 00:00:00 +0000

Introduction

There are many great things in base R, one of them is the tools package. This package has the functions that are used to build, check and create packages, documentation and manuals.

As I wanted to know how CRAN works and its changes I was looking into the source code of tools. I found some internal functions that access freely available files with information about CRAN packages. These private functions are at the CRANtools.R file.

packages <- tools::CRAN_package_db()
# current <- tools:::CRAN_current_db()
# archive <- tools:::CRAN_archive_db()
# issues <- tools::CRAN_check_issues()
# alias <- tools:::CRAN_aliases_db()
# rdxrefs <- tools:::CRAN_rdxrefs_db()

As I was not sure of the information on these files I asked on R-devel but I did not receive an answer. They seem to be quite obscure and as private functions might be removed without notice and shouldn’t be used in any dependency. However, as the files contain information about CRAN they might provide interesting clues about the history of CRAN and how it is operated.

On this post I will focus on the first file. I’ll explore a couple of fields and in future posts I will use the other files to explore more about CRAN history.

packages file

First of all a very brief exploration of what is in this file:

##    Package Version Priority                        Depends
## 1       A3   1.0.0     <NA> R (>= 2.15.0), xtable, pbapply
## 2 AATtools   0.0.1     <NA>                   R (>= 3.6.0)
## 3   ABACUS   1.0.0     <NA>                   R (>= 3.1.0)
##                                 Imports LinkingTo
## 1                                  <NA>      <NA>
## 2  magrittr, dplyr, doParallel, foreach      <NA>
## 3 ggplot2 (>= 3.1.0), shiny (>= 1.3.1),      <NA>
##                               Suggests Enhances    License License_is_FOSS
## 1                  randomForest, e1071     <NA> GPL (>= 2)            <NA>
## 2                                 <NA>     <NA>      GPL-3            <NA>
## 3 rmarkdown (>= 1.13), knitr (>= 1.22)     <NA>      GPL-3            <NA>

Packages has similar information as available.packages() but with many more columns with published date, reverse dependencies, X-CRAN-Comment, who packaged it… Also note that all this packages are not filtered to match R version, OS_type, subarch and there are almost duplicates (I learned about this filtering while reading the great documentation of available.packages() and also finding some mentions online).

As we have data from several years I’ll sometimes show the release dates of different R versions to provide some context. Without further delay let’s explore the data!

Published packages

CRAN started some time ago (in 1997) but it hasn’t remained frozen. The package archive (the A in CRAN) has been updating since then. For instance the current packages do not include packages that were removed, archived or those replaced by updates.

First packages are submitted to CRAN and once accepted they are published. As accepted and published usually are almost instantaneous I might use them as synonyms. Looking at the current available packages and their publication date, we can see the following:

Figure 1: Packages accepted on CRAN by the publication date.

The oldest package added was in 2010. This means a package without issues, dependencies changes, bugs detected by the automatic checks since 12 years!

The daily rate of acceptance has increased from less than 10 a day till 2020 to more than 30 this year 2022. If we summarize that information for month we see the same, but the little bump in 2020 disappears but we see other patterns:

Figure 2: Monthly packages published to CRAN. Some monthly variance is observed.

Instead of just one bump we see some waves with less packages on CRAN accepted late in the year and an increase of packages the first months of the year.

If we look at the accumulated packages on CRAN we see an exponential growth:

Figure 3: Acumulation of packages. Most of the packages have been published in the last 2 years.

In fact, most packages currently on CRAN where added since March 2021 than all the previous years.

Figure 4: Percentage of current packages on CRAN according to their date of publication. Most of them were published/updated on the last year and a half.

This is a good time to remind that the date being used is the date of publication of this version of the packages. Many had previous versions on CRAN:

First release	Packages
No	14,294
Yes	4,113

Processing time

Previously I found that CRAN submissions present some key differences between new packages and already published packages which impact how long do they need to wait to be published on CRAN. With the existing data we can compare how fast is the process by comparing the published date with the build date.

The build date is added to the tar.gz file automatically when the developer builds the package via R CMD build. However, the published date is set by CRAN once the packages are accepted on CRAN.

To visualize the differences I will also compare if there is some difference with new packages and those that were already on CRAN:

Figure 5: Histogram of time difference between building and publishing a package. Color indicates if the package is new to CRAN or not. Most of the published packages take more or less the same time regardless of if it is the first time or not.

There doesn’t seem to be much difference between date of building and date of publication according to if it is the first release or not. The precision is just a day and this is usually a fast process well below 50 days. Few packages exceed spend so much after build before publication and they are too few to be noticeable at this scale. Since 2016/05/02 there is a check that raises an issue if the build is older than a month.

Note that one might need to build multiple times the package before it is accepted. Packages published for the first time on CRAN might have been submitted previously, but when they finally built and pass the checks and manual review they are handled as fast as packages already on CRAN.

However, this time between build and acceptance might have changed with time:

Figure 6: Processing time between building the package and being published by date. There is a high difference between new packages and old ones. New packages usually take more time while existing packages take less than a day currently.

We clearly see a difference in processing time for those packages already on CRAN and those that are not. Keep in mind that for the few packages from before 2016 the estimation might not be accurate. At the same time this is consistent with the manual review process (For more information see my previous post about the review process of CRAN or my talk at the useR2021). It also means that there is a huge variation of time about how packages are handled. However this seems to be reducing: while in 2010 it took around 2 weeks, nowadays it takes less than a week and getting closer to a 1 day of median time between a package being built and appearing on CRAN that takes for existing packages.

This difference might be explainable due to experience: authors and maintainers whose package(s) are already in CRAN know better how to submit a new version without problems the checks.

It could also be that new packages need more time from the CRAN team. In 2020 we see it took longer than in previous years for packages to be added on CRAN. Maybe the increase in the processing time in 2020 was due the huge volume of submissions CRAN received or more checks on the developer side before submitting it to CRAN.

Both explanations are not mutually exclusive.

More packages published the same day mean more processing time? It doesn’t look like it.

Figure 7: Packages accepted the same day and processing time.New packages are accepted sooner than packages on CRAN respect to the builddate.

Surprisingly, we see a lot of variation on the delay of packages already accepted on CRAN. In addition, the more new packages accepted the same day, the less delay there is. I think this just means that when reviewers work on the submission queue several packages might be approved.

This might also mean packages have already been built several times before finally being accepted and now the errors, warnings and notes have been solved. Last, this could indicate that developers with their package already on CRAN wait a bit between building and submitting the package as the developer might be taking some time to double check before submission (dependencies, on several machines, other?) or a time zone difference (submitting in the noon of a region but at the reviewers night).

Conclusion

There are packages that for 12 years have been working without problems despite the several major changes in R (See figure 1). This speaks volumes of the packages’ quality, and the backward compatibility that the R core aims and CRAN checks.
CRAN accepts an incredible amount of packages daily and monthly. The system and the team are doing an incredible work mostly on their free time (See figure 2). Many thanks!
Accepted packages are handled very fast, in less than a week usually (See figure 7). But it is not possible to distinguish alone time in the submission system and time on the developer computer.

Future parts

We’ve explored a snapshot of current packages and a brief window of all the history of CRAN. There is much more that can be done with all the other files.

On future posts I’ll explore:

patterns accepting packages and updates in packages.
who handled the packages.
Size of packages.
the relation between dependencies, initial release and updates.

Other suggestions?

Edit: Many thanks to Maëlle Salmon and Dirk Eddelbuettel for their feedback on an initial version of this series of posts.

Reproducibility

## - Session info -------------------------------------------------------------------------------------------------------
##  setting  value
##  version  R version 4.2.1 (2022-06-23)
##  os       Ubuntu 20.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  C
##  ctype    C
##  tz       Europe/Madrid
##  date     2022-07-23
##  pandoc   2.18 @ /usr/lib/rstudio/bin/quarto/bin/tools/ (via rmarkdown)
## 
## - Packages -----------------------------------------------------------------------------------------------------------
##  package      * version    date (UTC) lib source
##  assertthat     0.2.1      2019-03-21 [2] RSPM (R 4.2.0)
##  base64enc      0.1-3      2015-07-28 [2] CRAN (R 4.0.0)
##  blogdown       1.10       2022-05-10 [2] RSPM (R 4.2.0)
##  bookdown       0.27       2022-06-14 [2] RSPM (R 4.2.0)
##  bslib          0.4.0      2022-07-16 [2] RSPM (R 4.2.0)
##  cachem         1.0.6      2021-08-19 [2] RSPM (R 4.2.0)
##  cli            3.3.0      2022-04-25 [2] RSPM (R 4.2.0)
##  codetools      0.2-18     2020-11-04 [2] RSPM (R 4.2.0)
##  colorspace     2.0-3      2022-02-21 [2] RSPM (R 4.2.0)
##  crayon         1.5.1      2022-03-26 [2] RSPM (R 4.2.0)
##  curl           4.3.2      2021-06-23 [2] RSPM (R 4.2.0)
##  data.table     1.14.2     2021-09-27 [2] RSPM (R 4.2.0)
##  DBI            1.1.3      2022-06-18 [2] RSPM (R 4.2.0)
##  digest         0.6.29     2021-12-01 [2] RSPM (R 4.2.0)
##  dplyr        * 1.0.9      2022-04-28 [2] RSPM (R 4.2.0)
##  ellipsis       0.3.2      2021-04-29 [2] RSPM (R 4.2.0)
##  evaluate       0.15       2022-02-18 [2] RSPM (R 4.2.0)
##  fansi          1.0.3      2022-03-24 [2] RSPM (R 4.2.0)
##  farver         2.1.1      2022-07-06 [2] RSPM (R 4.2.0)
##  fastmap        1.1.0      2021-01-25 [2] RSPM (R 4.2.0)
##  flextable    * 0.7.2      2022-06-12 [2] RSPM (R 4.2.0)
##  forcats      * 0.5.1      2021-01-27 [2] RSPM (R 4.2.0)
##  gdtools        0.2.4      2022-02-14 [2] RSPM (R 4.2.0)
##  generics       0.1.3      2022-07-05 [2] RSPM (R 4.2.0)
##  geomtextpath * 0.1.0      2022-01-24 [2] CRAN (R 4.2.1)
##  ggplot2      * 3.3.6.9000 2022-06-29 [2] Github (tidyverse/ggplot2@7571122)
##  ggrepel      * 0.9.1      2021-01-15 [2] RSPM (R 4.2.0)
##  glue           1.6.2      2022-02-24 [2] RSPM (R 4.2.0)
##  gtable         0.3.0      2019-03-25 [2] CRAN (R 4.0.0)
##  highr          0.9        2021-04-16 [2] RSPM (R 4.2.0)
##  htmltools      0.5.3      2022-07-18 [2] RSPM (R 4.2.0)
##  jquerylib      0.1.4      2021-04-26 [2] RSPM (R 4.2.0)
##  jsonlite       1.8.0      2022-02-22 [2] RSPM (R 4.2.0)
##  knitr          1.39       2022-04-26 [2] RSPM (R 4.2.0)
##  labeling       0.4.2      2020-10-20 [2] RSPM (R 4.2.0)
##  lattice        0.20-45    2021-09-22 [3] CRAN (R 4.2.0)
##  lifecycle      1.0.1      2021-09-24 [2] RSPM (R 4.2.0)
##  lubridate    * 1.8.0      2021-10-07 [2] RSPM (R 4.2.0)
##  magrittr       2.0.3      2022-03-30 [2] RSPM (R 4.2.0)
##  Matrix         1.4-1      2022-03-23 [2] RSPM (R 4.2.0)
##  mgcv           1.8-40     2022-03-29 [2] RSPM (R 4.2.0)
##  munsell        0.5.0      2018-06-12 [2] RSPM (R 4.2.0)
##  nlme           3.1-158    2022-06-15 [2] RSPM (R 4.2.0)
##  officer        0.4.3      2022-06-12 [2] RSPM (R 4.2.0)
##  pillar         1.8.0      2022-07-18 [2] RSPM (R 4.2.0)
##  pkgconfig      2.0.3      2019-09-22 [2] RSPM (R 4.2.0)
##  purrr          0.3.4      2020-04-17 [2] RSPM (R 4.2.0)
##  R6             2.5.1      2021-08-19 [2] RSPM (R 4.2.0)
##  Rcpp           1.0.9      2022-07-08 [2] RSPM (R 4.2.0)
##  rlang          1.0.4      2022-07-12 [2] RSPM (R 4.2.0)
##  rmarkdown      2.14       2022-04-25 [2] RSPM (R 4.2.0)
##  rstudioapi     0.13       2020-11-12 [2] RSPM (R 4.2.0)
##  rversions    * 2.1.1      2021-05-31 [2] RSPM (R 4.2.0)
##  sass           0.4.2      2022-07-16 [2] RSPM (R 4.2.0)
##  scales         1.2.0      2022-04-13 [2] RSPM (R 4.2.0)
##  sessioninfo    1.2.2      2021-12-06 [2] RSPM (R 4.2.0)
##  stringi        1.7.8      2022-07-11 [2] RSPM (R 4.2.0)
##  stringr        1.4.0      2019-02-10 [2] RSPM (R 4.2.0)
##  systemfonts    1.0.4      2022-02-11 [2] RSPM (R 4.2.0)
##  textshaping    0.3.6      2021-10-13 [2] RSPM (R 4.2.0)
##  tibble         3.1.7      2022-05-03 [2] RSPM (R 4.2.0)
##  tidyr        * 1.2.0      2022-02-01 [2] RSPM (R 4.2.0)
##  tidyselect     1.1.2      2022-02-21 [2] RSPM (R 4.2.0)
##  utf8           1.2.2      2021-07-24 [2] RSPM (R 4.2.0)
##  uuid           1.1-0      2022-04-19 [2] RSPM (R 4.2.0)
##  vctrs          0.4.1      2022-04-13 [2] RSPM (R 4.2.0)
##  withr          2.5.0      2022-03-03 [2] RSPM (R 4.2.0)
##  xfun           0.31       2022-05-10 [2] RSPM (R 4.2.0)
##  xml2           1.3.3      2021-11-30 [2] RSPM (R 4.2.0)
##  yaml           2.3.5      2022-02-21 [2] RSPM (R 4.2.0)
##  zip            2.2.0      2021-05-31 [2] RSPM (R 4.2.0)
## 
##  [1] /home/lluis/bin/R/4.2.1
##  [2] /usr/lib/R/site-library
##  [3] /usr/lib/R/library
## 
## ----------------------------------------------------------------------------------------------------------------------

Upgrading rtweet to 1.0.2

Mon, 04 Jul 2022 00:00:00 +0000

In this post I will provide some examples of what has changed between rtweet 0.7.0 and rtweet 1.0.2. I hope both the changes and this guide will help all users. I highlight the most important and interesting changes in this blog post, and for a full list of changes you can consult it on the NEWS.

Big breaking changes

More consistent output

This is probably what will affect the most users. All functions that return data about tweets¹ now return the same columns.

For example if we search some tweets we’ll get the following columns:

> tweets <- search_tweets("weather")
> colnames(tweets)
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "full_text"                    
 [5] "truncated"                     "display_text_range"           
 [7] "entities"                      "metadata"                     
 [9] "source"                        "in_reply_to_status_id"        
[11] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[13] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[15] "geo"                           "coordinates"                  
[17] "place"                         "contributors"                 
[19] "is_quote_status"               "retweet_count"                
[21] "favorite_count"                "favorited"                    
[23] "retweeted"                     "lang"                         
[25] "quoted_status_id"              "quoted_status_id_str"         
[27] "quoted_status"                 "possibly_sensitive"           
[29] "retweeted_status"              "text"                         
[31] "favorited_by"                  "scopes"                       
[33] "display_text_width"            "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "query"                         "withheld_scope"               
[41] "withheld_copyright"            "withheld_in_countries"        
[43] "possibly_sensitive_appealable"

rtweet now minimizes the processing of tweets and only returns the same data as provided by the API while making it easier to handle by R. However, to preserve the nested nature of the data returned some fields are now nested inside other. For example, previously fields "bbpx_coords", "geo_coords", "coords_coords" were returned as separate columns, but they are now nested inside "place", "coordinates" or "geo" depending where they are provided. Some columns previously calculated by rtweet are now not returned, like "rtweet_favorite_count". At the same time it provides with new columns about each tweet like the "withheld_*" columns.

If you scanned through the columns you might have noticed that columns "user_id" and "screen_name" are no longer returned. This data is still returned by the API but it is now made available to the rtweet users via users_data():

> colnames(users_data(tweets))
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

This blog post should help you find the right data columns, but if you don’t find what you are looking for it might be nested inside a column.
Try using dplyr::glimpse() to explore the data and locate nested columns. For example the entities column (which is present in both tweets and users) have the following useful columns:

> names(tweets$entities[[1]])
[1] "hashtags"      "symbols"       "user_mentions" "urls"         
[5] "media"

Similarly if you look up a user via search_users() or lookup_users() you’ll get consistent data:

> users <- lookup_users(c("twitter", "rladiesglobal", "_R_Foundation"))
> colnames(users)
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

You can use tweets_data() to retrieve information about their latest tweet:

> colnames(tweets_data(users))
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "text"                         
 [5] "truncated"                     "entities"                     
 [7] "source"                        "in_reply_to_status_id"        
 [9] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[11] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[13] "geo"                           "coordinates"                  
[15] "place"                         "contributors"                 
[17] "is_quote_status"               "retweet_count"                
[19] "favorite_count"                "favorited"                    
[21] "retweeted"                     "lang"                         
[23] "retweeted_status"              "possibly_sensitive"           
[25] "quoted_status"                 "display_text_width"           
[27] "user"                          "full_text"                    
[29] "favorited_by"                  "scopes"                       
[31] "display_text_range"            "quoted_status_id"             
[33] "quoted_status_id_str"          "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "metadata"                      "query"                        
[41] "withheld_scope"                "withheld_copyright"           
[43] "withheld_in_countries"         "possibly_sensitive_appealable"

You can merge them via:

users_and_last_tweets <- cbind(users, id_str = tweets_data(users)[, "id_str"])

In the future (see below), with helper functions managing the output of rtweet will become easier.

Finally, get_followers() and get_friends() now return the same columns:

> colnames(get_followers("_R_Foundation"))
[1] "from_id" "to_id"  
> colnames(get_friends("_R_Foundation"))
[1] "from_id" "to_id"

This will make it easier to build networks of connections (although you might want to convert screen names to ids or vice versa).

More consistent interface

All paginated functions that don’t return tweets now use a consistent pagination interface (except the premium endpoints). They all store the “next cursor” in an rtweet_cursor attribute, which will be automatically retrieved when you use the cursor argument. This will make it easier to continue a query you started:

users <- get_followers("_R_Foundation")
users
     
# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

They support max_id and since_id to find earlier and later tweets respectively:

# Retrieve all the tweets made since the previous request
newer <- search_tweets("weather", since_id = tweets)
# Retrieve tweets made before the previous request
older <- search_tweets("weather", max_id = tweets)

If you want more tweets than it is allowed by the rate limits of the API, you can use retryonratelimit to wait as long as needed:

long <- search_tweets("weather", n = 1000, retryonratelimit = TRUE)

This will keep busy your terminal until the 1000 tweets are retrieved.

Saving data

An unexpected consequence of returning more data (now matching that returned by the API) is that it is harder to save it in a tabular format. For instance one tweet might have one media, mention two users and have three hashtags. There isn’t a simple way to save it in a single row uniformly for all tweets or it could lead to confusion.

This resulted in deprecating save_as_csv, read_twitter_csv and related functions because they don’t work with the new data structure and it won’t be possible to load the complete data from a csv. They will be removed in later versions.

Many users will benefit from saving to RDS (e.g., saveRDS() or readr::write_rds()), and those wanting to export to tabular format can simplify the data to include only that of interest before saving with generic R functions (e.g., write.csv() or readr::write_csv()).

Other breaking changes

Accessibility is important and for this reason if you tweet via post_tweet() and add an image, gif or video you’ll need to provide the media alternative text. Without media_alt_text it will not allow you to post.
tweet_shot() has been deprecated as it no longer works correctly. It might be possible to bring it back, but the code is complex and I do not understand enough to maintain it. If you’re interested in seeing this feature return, checkout the discussion about this issue and let me know if you have any suggestions.
rtweet also used to provide functions for data on emojis, langs and stopwordslangs. These are useful resources for text mining in general - not only in tweets - however they need to be updated to be helpful and would be better placed in other packages, for instance emojis is now on the bdpar package. Therefore they are no longer available in rtweet.
The functions like suggested_*() have been removed as they have been broken since 2019.

Easier authentication

An exciting part of this release has been a big rewrite of the authentication protocol. While it is compatible with previous rtweet authentication methods it has also some important new functions which make it easier to work with rtweet and the twitter API in different ways.

Different ways to authenticate

If you just want to test the package, use the default authentication auth_setup_default() that comes with rtweet. If you use it for one or two days you won’t notice any problem.

If you want to use the package for more than a couple of days, I recommend you set up your own token via rtweet_user(). It will open a window to authenticate via the authenticated account in your default browser. This authentication won’t allow you to do everything but it will avoid running out of requests and being rate-limited.

If you plan to make heavy use of the package, I recommend registering yourself as developer and using one of the following two mechanisms, depending on your plans:

Collect data and analyze: rtweet_app().
Set up a bot: rtweet_bot()

Find more information in the Authentication with rtweet vignette.

Storing credentials

Previously rtweet saved each token created, but now non-default tokens are only saved if you ask. You can save them manually via auth_save(token, "my_app"). Bonus, if you name your token as default (auth_save(token, "default")) it will be used automatically upon loading the library.

Further, tokens are now saved in the location output by tools::R_user_dir("rtweet", "config"), rather than in your home directory. If you have previous tokens saved or problems identifying which token is which use auth_sitrep(). This will provides clues to which tokens might be duplicated or misconfigured but it won’t check if they work. It will also automatically move your tokens to the new path.

To check which credentials you have stored use auth_list() and load them via auth_as("my_app"). All the rtweet functions will use the latest token loaded with auth_as (unless you manually specify one when calling it). If you are not sure which token you are using you can use auth_get() it will return the token in use, list them or ask you to authenticate.

Other changes of note

This is a list of other changes that aren’t too big or are not breaking changes but are worthy enough of a mention:

Iteration and continuation of requests

Using cursors, pagination or waiting until you can make more queries is now easier. For example you can continue previous requests via:

users <- get_followers("_R_Foundation")
users

# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

Additions

There is now a function to find a thread of a user. You can start from any tweet and it will find all the tweets of the thread: tweet_threading("1461776330584956929").

There is a lot of interest in downloading and keeping track of interactions on Twitter. The amount of interest is big enough that Twitter is releasing a new API to provide more information of this nature.

Future

Twitter API v2 is being released and soon it will replace API v1. rtweet up to now, including this release, uses API v1 so it will need to adapt to the new endpoints and new data returned.

First will be the streaming endpoints in November, so expect more (breaking?) changes around those dates if not earlier.

I would also like to make it easier for users, dependencies and the package itself to handle the outputs. To this regard I would like to provide some classes to handle the different type of objects it returns.

This will help avoid some of the current shortcomings. Specifically I would like to provide functions to make it easier to reply to previous tweets, extract nested data, and subset tweets and the accompanying user information.

Conclusions

While I made many breaking changes I hope these changes will smooth future development and help both users and maintainers.

Feel free to ask on the rOpenSci community if you have questions about the transition or find something amiss. Please let me know! It will help me prioritize which endpoints are more relevant to the community. (And yes, the academic archive endpoint is on the radar.)

It is also possible that I overlooked something and I thought the code is working when it isn’t. For example, after several months of changing the way the API is parsed, several users found it wasn’t handling some elements. Let me know of such or similar cases and I’ll try to fix it.

In case you find a bug, check the open issues and if it has not already been reported, open an issue on GitHub. Don’t forget to make a reprex and if possible provide the id of the tweets you are having trouble with. Unfortunately it has happened that when I came to look at a bug I couldn’t reproduce it as I wasn’t able to find the tweet which caused the error.

This release includes contributions from Hadely Wicham, Bob Rudis, Alex Hayes, Simon Heß, Diego Hernán, Michael Chirico, Jonathan Sidi, Jon Harmon, Andrew Fraser and many other that reported bugs or provided feedback. Many thanks all for using it, your interest to keep it working and improving rtweet for all.

Finally, you can read the whole NEWS online and the examples.

Happy tweeting!

Acknowledgements

This is a repost of the entry for rOpenSci. The post was edited and improved by Yanina Bellini Saibene and Steffi LaZerte, the community manager and assistant. Many thanks

Specifically these: get_favorites(), get_favorites_user(), get_mentions(), get_my_timeline(), get_retweets(), get_timeline(), get_timeline_user(), lists_statuses(), lookup_statuses(), lookup_tweets(), search_30day(), search_fullarchive(), search_tweets(), tweet_shot() and tweet_threading().↩︎

Creating an RSS feed for r-bloggers for my blogdown

Thu, 30 Dec 2021 00:00:00 +0000

I am following Yongfu’s blog: I copy here the content in case it is moved:

Copy the default RSS template

Change {{ .Summary | html }} to {{ .Content | html }}

Create subdirectories in layouts/ and save the RSS template files (name them rss.xml) in them (one for each subdirectory).

I was first confused when I wanted to copy the rss.xml file mentioned and didn’t know where to go. I finally found the section from which I could go to the file I was supposed to copy.

So far so good, so without further testing I assume this worked. Following the instructions, I moved the rss.xml file to layouts/post/ to only report posts of the blog, not new projects or other leafs from Hugo.

I also created a rss.xml on layouts/categories/. As the templates provided by gohugo.io didn’t work I copied and pasted the files from Yongfu’s Blog.

To check that it worked I went to llrs.dev/categories/cran/index.xml. It did but not for all the posts. I kept getting an error even when I set a limit of 20 post on the config.toml file rssLimit = 20.

I abandoned this for a while, other projects, pandemia…

When I came back I updated the template on 30/12/2021 and the same error was still happening. After some digging I found that there is a bug on Hugo generating invalid XML code for the feeds.

I started looking for workarounds but adding a php function or a go function didn’t work well. I couldn’t correctly set up the php function and I don’t know how to write go.

Then I started thinking on limiting the issue, which let it to rssLimit. After several checks I didn’t manage to make it work.

But while reading a post I realized that rssLimit was no longer used (if it was ever on the first place).

So after adding:

[Services.RSS]
  limit = 10

on my config.toml with the new template, the xml feeds are working again. There is only one problem. The twitter tag feed doesn’t work, because the post that creates invalid content is there.

Hope that this save people some time if they try to do that. If you follow the linked posts like this one and take into account this change on how to limit RSS you might avoid the problem.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.2.0 (2022-04-22)
##  os       Ubuntu 20.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2022-05-09
##  pandoc   2.17.1.1 @ /usr/lib/rstudio/bin/quarto/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date (UTC) lib source
##  blogdown      1.9     2022-03-28 [1] CRAN (R 4.2.0)
##  bookdown      0.26    2022-04-15 [1] CRAN (R 4.2.0)
##  bslib         0.3.1   2021-10-06 [1] CRAN (R 4.2.0)
##  cli           3.3.0   2022-04-25 [1] CRAN (R 4.2.0)
##  digest        0.6.29  2021-12-01 [1] CRAN (R 4.2.0)
##  evaluate      0.15    2022-02-18 [1] CRAN (R 4.2.0)
##  fastmap       1.1.0   2021-01-25 [1] CRAN (R 4.2.0)
##  htmltools     0.5.2   2021-08-25 [1] CRAN (R 4.2.0)
##  jquerylib     0.1.4   2021-04-26 [1] CRAN (R 4.2.0)
##  jsonlite      1.8.0   2022-02-22 [1] CRAN (R 4.2.0)
##  knitr         1.39    2022-04-26 [1] CRAN (R 4.2.0)
##  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.2.0)
##  R6            2.5.1   2021-08-19 [1] CRAN (R 4.2.0)
##  rlang         1.0.2   2022-03-04 [1] CRAN (R 4.2.0)
##  rmarkdown     2.14    2022-04-25 [1] CRAN (R 4.2.0)
##  rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.2.0)
##  sass          0.4.1   2022-03-23 [1] CRAN (R 4.2.0)
##  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.2.0)
##  stringi       1.7.6   2021-11-29 [1] CRAN (R 4.2.0)
##  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.2.0)
##  xfun          0.30    2022-03-02 [1] CRAN (R 4.2.0)
##  yaml          2.3.5   2022-02-21 [1] CRAN (R 4.2.0)
## 
##  [1] /home/lluis/bin/R/4.2.0/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Reasons why packages are archived on CRAN

Tue, 07 Dec 2021 00:00:00 +0000

On the Repositories working group of the R Consortium Rich FitzJohn posted a comment to a file that seems to be were the CRAN team stores and uses to check the package history.

The structure is not defined anywhere I could find (I haven’t looked much to be honest).

Package: <package name>
X-CRAN-Comment: Archived on YYYY-MM-DD as <reason>.
X-CRAN-History: Archived on YYYY-MM-DD as <reason>.
  Unarchived on YYYY-MM-DD.
  .
  <Optional clarification of archival reason>
<Optional fields like License_restricts_use, Replaced_by, Maintainer: ORPHANED, OS_type: unix>

I think the X-CRAN-Comment is what appears on the website of an archived package, like on radix package. However, other comments on the website do not appear on that file.

In addition, the file doesn’t have some records of archiving and unarchiving of some packages, but there are old records from 2013 or before to now. But we can use it to see understand what are the reasons of archiving packages, which seems to be the main purpose of the file.

The data

First step is read the record. As it seems that it has some key: value structure similar to DESCRIPTION file of packages it seems it is a DCF format: Debian Control File format which is easy to read with the read.dcf function.

Exploring

A brief exploration of the data:

comment	history	packages
yes	no	3612
no	yes	2345
yes	yes	434
no	no	70

Many packages have either comments or history but relatively few both. I’m not sure when either of them is used, as I would expect that all that have history would have a comment.

Replaced_by	packages
no	6360
yes	101

Many packages are simply replaced by some other package.

Maintainer	packages
no	6366
yes	95

Most of the packages that have a Maintainer field are orphaned/archived. Does it mean that all the others are not orphaned?

Extracting reasons

Now that it is in R data structure, we can extract the relevant information, dates, type of action and reasons for each archivation event. I use strcapture for this task with a regex to extract the action, the date and the explanation it migh have.

I don’t know how the file is written probably it is a mix of automated tools and manual editing so there isn’t a simple way to collect all the information in a structured way. Simply because the structure has been changing along the years as well as the details of what is stored has changed, or there are missing events. However, the extracted information should be enough for our purposes.

Action	Events
archived	7096
orphaned	341
removed	113
renamed	2
replaced	4
unarchived	2973

As expected the most common recorded event are archivations, but there are some orphaned packages and even some removed packages. Also note the number of orphaned packages is greater than those with the Maintainer field, supporting my theory that the format has changed and that this shouldn’t be taken as an exhaustive and complete analysis of archivations.

How are they along time?

Even if there are some events recorded from 2009 it seems that this file has been more used more recently (last commit related to this was on 2015). I know that there are some old events not recorded on the file, because there are some packages currently present on CRAN that they had been archived but do not have an unarchived action, so conversely it could happen. So, this doesn’t necessarily mean that there are currently more packages archived from CRAN. But it is a clear indication that now at least there is a more accurate record of archived packages on this file.

Another source of records of archived packages might be cranberries. It would be nice to compare this file with the records on the database there.

Now that most of the package events are collected and we have the reasons of the actions, we can explore and classify the reasons. Using some simple regex I explore for key words or sentences.

We can look at the most frequent error reasons for archiving packages, patterns I found with more than 100 cases:

The most frequent error is that errors are not corrected or checks, even when there are reminders.
Next are the packages archived because they depend on other packages already not on CRAN.
There are some packages that are replaced by others and some maintainers might not want to continue supporting the package when they receive a message from CRAN about fixing an error.

Policy violation makes to the top 5 but with less than 500 events. Dependencies problems are the sixth cause, followed by email errors (bouncing, incorrect email…) and then come very sporadic problems about license, not fixing on updates of R, authorship problems or requests from authors.

Some of these errors happen at the same time for each event, but grouping these reasons together we get a similar table:

package_not_corrected	request_maintainer	dependencies	other	events
yes	no	no	no	4366
no	no	no	no	1530
no	no	yes	no	767
no	no	no	yes	374
yes	no	no	yes	15
yes	no	yes	no	13
no	no	yes	yes	2
yes	no	yes	yes	2
yes	yes	no	no	2
yes	yes	no	yes	1

Surprisingly the second most frequent group of archiving actions are due to many different reasons. This is probably the Pareto’s principle in action because they are around 15% of the archiving events but the causes are very diverse between them.

However, if we look at the packages which were archived (not at the request of maintainers), most of them just happen once:

Events	packages
1	5304
2	594
3	115
4	31
5	8

This suggests that once a package is archived maintainers do not make the effort to put it back on CRAN except on very few cases were there are multiple attempts. To check we can see the current available packages and see how many of those are still present on CRAN:

CRAN	Packages	Proportion
no	3869	64%
yes	2183	36%

Many packages are currently on CRAN despite their past archivation but close to 64% are currently not on CRAN.

Almost all that are on CRAN have now no X-CRAN-Comment, except for a few:

Package	X-CRAN-Comment
geiger	Orphaned and corrected on 2022-05-09. Repeated notifications about USE_FC_LEN_T were ignored.
alphahull	Versions up to 2.3 have been removed for mirepresentation of authorship.
udunits2	Orphaned on 2022-01-06 as installation problems were not corrected.
bibtex	Orphaned and corrected on 2020-09-19 as check problems were not corrected in time.

CRAN team might have missed these few packages and didn’t move the comments to X-CRAN-history.

There are some packages that are not archived that don’t have a CRAN-history happens too, but they usually have other fields changed.

Discussion

Most packages archived on CRAN are due to the maintainers not correcting errors found on the package by CRAN checks. It is clear that the checks that CRAN help packages to have a high quality but it has high cost on the maintainers and specially on CRAN team. Maintainers don’t seem to have enough time to fix the issues on time. And the CRAN team sends personalized reminders to maintainers and sometimes patches to the packages.

Although the desire to have packages corrected and with no issues is the common goal there are few options on light of these:

Be more restrictive

Prevent a package to be accepted if it breaks dependencies or archive packages when they fail checks. This will make it harder to keep packages on CRAN but would lift some pressure on the CRAN team. This would go against the current on other languages repositories, which often they don’t check the packages/modules and even have less restrictions on dependencies (so it might be an unpopular decision).
Be more permissive:

One option would be to allow for more time for maintainers to fix issues. I haven’t find any report of how long does it take for a package since an error to a fix on CRAN but often it is quite long. I have seen packages with a warning for months if not years and they weren’t archived from CRAN.

Maybe if users get a warning on installing packages that a package or one of its dependencies is not clear on all CRAN checks (without error or warnings). This might help to make users more conscious of their dependencies but this might add pressure to maintainers who already don’t have enough time to fix the problems of their packages.
Provide more help or tools to maintainers

Another option is to provide a mechanism for maintainers to receive help or fix the package. Currently CRAN requires that new packages that break dependencies to give enough notice in advance to other maintainers to fix their package. On R-pkg-devel mailing list there are often requests for help on submitting and fixing some errors detected by CRAN checks which often result on other maintainers sharing their solutions for the same problem.

There high percentage of packages that once archived do not come back to CRAN might be a good place to start helping maintainers and an opportunity for users to step in and help maintainers of packages they have been using. There is need for something else? How would that work?

At the same time it is admirable that after so many years there are few errors on the data. However, the archival process might be a good process to automate, providing the reason on the webpage and add it to X-CRAN-Comment and moving the comments to X-CRAN-History once it is unarchived. Knowing more about how these actions are performed by the CRAN team and how the community could help on the process will be beneficial to all.

Note: This blog was updated on 2022/01/02 to improve the parsing of actions and dates on packages. Resulting on a change on the first plot to include unarchived which slightly modified the second plot of reasons why packages are archived. This overall only affected the numbers of the plots not the conclusions or discussion.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.2.0 (2022-04-22)
##  os       Ubuntu 20.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2022-05-09
##  pandoc   2.17.1.1 @ /usr/lib/rstudio/bin/quarto/bin/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package      * version date (UTC) lib source
##  assertthat     0.2.1   2019-03-21 [1] CRAN (R 4.2.0)
##  blogdown       1.9     2022-03-28 [1] CRAN (R 4.2.0)
##  bookdown       0.26    2022-04-15 [1] CRAN (R 4.2.0)
##  bslib          0.3.1   2021-10-06 [1] CRAN (R 4.2.0)
##  cli            3.3.0   2022-04-25 [1] CRAN (R 4.2.0)
##  colorspace     2.0-3   2022-02-21 [1] CRAN (R 4.2.0)
##  ComplexUpset * 1.3.3   2021-12-11 [1] CRAN (R 4.2.0)
##  crayon         1.5.1   2022-03-26 [1] CRAN (R 4.2.0)
##  DBI            1.1.2   2021-12-20 [1] CRAN (R 4.2.0)
##  digest         0.6.29  2021-12-01 [1] CRAN (R 4.2.0)
##  dplyr        * 1.0.9   2022-04-28 [1] CRAN (R 4.2.0)
##  ellipsis       0.3.2   2021-04-29 [1] CRAN (R 4.2.0)
##  evaluate       0.15    2022-02-18 [1] CRAN (R 4.2.0)
##  fansi          1.0.3   2022-03-24 [1] CRAN (R 4.2.0)
##  farver         2.1.0   2021-02-28 [1] CRAN (R 4.2.0)
##  fastmap        1.1.0   2021-01-25 [1] CRAN (R 4.2.0)
##  generics       0.1.2   2022-01-31 [1] CRAN (R 4.2.0)
##  ggplot2      * 3.3.6   2022-05-03 [1] CRAN (R 4.2.0)
##  glue           1.6.2   2022-02-24 [1] CRAN (R 4.2.0)
##  gtable         0.3.0   2019-03-25 [1] CRAN (R 4.2.0)
##  highr          0.9     2021-04-16 [1] CRAN (R 4.2.0)
##  htmltools      0.5.2   2021-08-25 [1] CRAN (R 4.2.0)
##  jquerylib      0.1.4   2021-04-26 [1] CRAN (R 4.2.0)
##  jsonlite       1.8.0   2022-02-22 [1] CRAN (R 4.2.0)
##  knitr          1.39    2022-04-26 [1] CRAN (R 4.2.0)
##  labeling       0.4.2   2020-10-20 [1] CRAN (R 4.2.0)
##  lifecycle      1.0.1   2021-09-24 [1] CRAN (R 4.2.0)
##  magrittr       2.0.3   2022-03-30 [1] CRAN (R 4.2.0)
##  munsell        0.5.0   2018-06-12 [1] CRAN (R 4.2.0)
##  patchwork      1.1.1   2020-12-17 [1] CRAN (R 4.2.0)
##  pillar         1.7.0   2022-02-01 [1] CRAN (R 4.2.0)
##  pkgconfig      2.0.3   2019-09-22 [1] CRAN (R 4.2.0)
##  purrr          0.3.4   2020-04-17 [1] CRAN (R 4.2.0)
##  R6             2.5.1   2021-08-19 [1] CRAN (R 4.2.0)
##  rlang          1.0.2   2022-03-04 [1] CRAN (R 4.2.0)
##  rmarkdown      2.14    2022-04-25 [1] CRAN (R 4.2.0)
##  rstudioapi     0.13    2020-11-12 [1] CRAN (R 4.2.0)
##  sass           0.4.1   2022-03-23 [1] CRAN (R 4.2.0)
##  scales         1.2.0   2022-04-13 [1] CRAN (R 4.2.0)
##  sessioninfo    1.2.2   2021-12-06 [1] CRAN (R 4.2.0)
##  stringi        1.7.6   2021-11-29 [1] CRAN (R 4.2.0)
##  stringr        1.4.0   2019-02-10 [1] CRAN (R 4.2.0)
##  tibble         3.1.7   2022-05-03 [1] CRAN (R 4.2.0)
##  tidyselect     1.1.2   2022-02-21 [1] CRAN (R 4.2.0)
##  utf8           1.2.2   2021-07-24 [1] CRAN (R 4.2.0)
##  vctrs          0.4.1   2022-04-13 [1] CRAN (R 4.2.0)
##  withr          2.5.0   2022-03-03 [1] CRAN (R 4.2.0)
##  xfun           0.30    2022-03-02 [1] CRAN (R 4.2.0)
##  yaml           2.3.5   2022-02-21 [1] CRAN (R 4.2.0)
## 
##  [1] /home/lluis/bin/R/4.2.0/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────