r | B101nfo

Packaging R: getting in repositories

Sun, 05 May 2024 00:00:00 +0000

After the previous post collecting information about repositories I want to collect here my thoughts on adding a package in a repository and how repositories are recognized. As in the previous post this is built on the assumption that one already has a package or more and wants to distribute it.

This is meant as a reflection of what is an R repository and not intended for R package developers. However, their feedback is appreciated to consider how an ideal repository would be.

Package submission

An R repository will have a way to incorporate a package. CRAN submission process starts with a form, while Bioconductor is done through a Github issue.

The process will usually then start with an automated process. Until the automated process check hasn’t passed probably no one will look into the package submission. This reduce the hours a human must dedicate to manage submissions. If a man is kept in the loop one could appeal the automatic process contacting them, or if it is a random failing re-submitting the package again.

Package submission checks: first a check of the package, if it is not new a dependency check from the repository if all checks pass the package is added to the repository.

Generally a package must first pass a package quality check before it is considered for further integration test. This integration test is usually checking the new version of a package with packages that depend on it, also known as reverse dependencies.

Package maintenance

Once a package is included in a repository it usually needs to be maintained.

There are many moving pieces, chips architecture, OS, R, other packages. This all lead that authors need to maintain the packages in good shape if they want it to remain useful to users. Of course, if one doesn’t want to do that they do not need to create a repository to share their package.

Graphic showing time and different R versions and checks. Repositories check the packages on them on multiple levels.

This leads that at any given time point there must be some tests for any given package under different conditions as shown in image 2. This leads to the possibility of having a package archived from the repository for failing the checks in place.

Repositories provide these checks as a service to the users. They guarantee that R packages in the repository work well together and pass the same set of packages (mostly). This is what leads to their reputation and usage among users (this is true beyond R, DEBIAN, Ubuntu, …).

Closing remarks

There are several official repositories how the package submission works when a package is submitted to one but it is related, via dependencies to other repositories is a matter of another post.

There are some discussion on what is an R repository. The importance of CRAN and Bioconductor has lead to some confusion. There are generally two meanings of what a cran-like repository is:

One where install.packages() works (This is defined by how the files and binaries are organized and will be a theme for another time).
One were all the checks described here are in place and install.packages() works too.

r-universe is using the first definition but could be used to generate repositories with checks that comply with the second definition. Other repositories that use that are the Posit Public Package Manager, or the R4Pi repository (which provides binaries for Raspberry Pi OS).

As the second definition is more strict I’ll focus on it as this post has explained.

PS: This post might be edited as it has been siting in my computer for several months. I prefer to post it and be improved with feedback, so let me know if you have any addition.

useR madrid: rtweet

Thu, 29 Feb 2024 19:00:00 +0200

This presentation was in Spanish. I shared the history of my involvement with rtweet and what is happening with the package and Twitter API.

Packaging R: repositories

Sat, 09 Dec 2023 00:00:00 +0000

In this post I want to collect some thoughts about R repositories. In R we have multiple repositories that store packages for users. In this post I want to write about the purpose, functionality, benefits and drawbacks of R repositories and how packages are managed. The goal is to summarize what I’ve learnt these last years about them. I’ll also collect some information about them from various sources to make it easier for myself to find it later on.

I am writing this because I am worried about the future of CRAN and R. Due to multiple circumstances, the current position is not sustainable as is. I hope that this post, will help me to understand the past, present and create some concrete steps to do.

History

I was not there, but the first repository started around April 1997. This repository is CRAN: the Comprehensive R Archive Network. The first mention I found is already about changes in it, but it was not until the end of the month when it was announced.

CRAN was created by a few volunteers, some of which are still mainting it 25 years later. The current team is listed on their website. From the beginning it was “a collection of sites which carry identical material, consisting of the R&R R distribution(s), the contributed extensions, documentation for R, and binaries.”

Omegahat was another repository created shortly after CRAN:

The Omega project began in July, 1998, with discussions among designers responsible for three current statistical languages (S, R, and Lisp-Stat), with the idea of working together on new directions with special emphasis on web-based software, Java, the Java virtual machine, and distributed computing.

Many developers of Omegahat were in the R Core or CRAN team. It was available as a repository from the R source code but was removed definitely in version R 4.1, in 2021¹.

Bioconductor, was the next major repository that appeared. It was funded by Robert Gentleman and others in 2004 (it started the mailing list). A paper describing it appeared in late 2004:

an initiative for the collaborative creation of extensible software for computational biology and bioinformatics.

Through its history repositories have evolved with R and R with them. For example: R was released twice a year at the beginning, and Bioconductor did too. But when R moved to be released once per year (in 2013 with version 3.0) Bioconductor kept using two releases a year. This introduced some problems when installing packages from Bioconductor, when a single R release can be compatible with two Bioconductor releases².

In other cases, checks have evolved. For instance Solaris was used to test packages in CRAN until 2021, if I recall correctly, because it allowed to test in a proprietary C or C++ compiler. This lead to discover bugs but also to more distress in R-package developers which had difficulties checking their packages in that environment.

Other checks evolve with R, becoming more strict with time: In the early versions of R the use of NAMESPACE was not regulated. But since R version 2.15, 2012 it was compulsory even for data-only packages³. This was synchronized with repositories checks.

Last, some goals/desires of CRAN are not fulfilled (or where abandoned). For example, from the start CRAN aimed to have packages authenticated (see the bottom of the announcement). This might be due to lack of time, resources or that the plans are in progress but require (volunteer) time.

With time, different repositories arose:

MRAN, which was available since September 17th, 2014 to July 1st, 2022.
The Rstudio Public Package Manager later renamed Posit Public Package Manager has binaries for several OS since 2019.
There is the R4pi repository with binaries for Raspberry Pi.
I remember a proteomics repository available.
rOpenSci started its own repository which later evolved into the r-universe. The r-universe currently can provide binaries of packages that are hosted in a git repository.

Literature

The role and prominence of the repositories has lead to many articles being written about it. I wanted to link and collect some of them for easier retrieval.

I was wondering how CRAN is described by the volunteers that built it. From the announcing email:

CRAN is a collection of sites which carry identical material, consisting of the R&R R distribution(s), the contributed extensions, documentation for R, and binaries.

From the website (at 2023/12/09):

CRAN is a network of ftp and web servers around the world that store identical, up-to-date, versions of code and documentation for R.

Initially there was R NEWS, with an article dedicated to CRAN and one to Omegahat too. These articles usually describe new package additions but sometimes they also provide information about changes:

CRAN-2001-1: It list new packages, CRAN-2001-2, CRAN-2001-3, Omegahat-2001-3
CRAN-2002-1, CRAN-2002-2, CRAN-2002-3
CRAN-2003-1, CRAN-2003-2, CRAN-2003-3
CRAN-2004-1, CRAN-2004-2
CRAN-2005-1, CRAN-2005-2

Since 2006 there is also an article about Bioconductor.
CRAN-2006-2, Bioc-2006-2
CRAN-2007-1, CRAN-2007-2, Bioc-2007-2, CRAN-2007-3
CRAN-2008-1, Bioc-2008-1 CRAN-2008-2, Bioc-2008-2

Later it became the R Journal:

CRAN-2009-1, CRAN-2009-2
CRAN-2010-1, CRAN-2010-2
CRAN-2011-1, CRAN and Bioconductor 2011-2. In the bioconductor section it mentions conference, and important directions for the Bioconductor core.
CRAN-2012-1, CRAN and Bioconductor 2012-2: Mentions biocLite() to install packages.
CRAN-2013-1 Bioc-2013-1: mentions better integration of parallel evaluation.
CRAN-2013-2, Bioc-2013-2: Mentions again AnnotationHub
CRAN-2014-1, Bioc-2014-1: Mentions the git-svn bridge to synchronize git and svn repository.
CRAN-2014-2, Bioc-2014-2: Bioconductor 3.0 release, besides some packages Amazon Machine Image are offered as well as docker images. Packages are required to pass BiocCheck, checks in a different package specific for Bioconductor.
CRAN-2015-1, Bioc-2015-1: Same mentions as the previous and encouragement to guidelines an package submission.
CRAN-2015-2, Bioc-2015-2
CRAN-2016-1: on this article there is a plot of the number of CRAN packages and time, and doesn’t list all packages listed. It explicitly mentions that the CRAN team asked for help processing package submissions and some people stepped up. Bioc-2016-1

CRAN-2016-2, Bioc-2016-2
CRAN-2017-1: mentions changes in CRAN checks, adding new memory access and static code analysis checks. It mentions that the submission has moved to a more automated one. It also mentions changes in the CRAN Repository Policy. Bioc-2017-1
CRAN-2018-1: checks in alternative BLAS/LAPACK implementations, the submission pipeline is defined. First time the amount of action taken by CRAN reviewers is listed in two categories automatic and manual. Changes in repository policy are listed. Changes in location of package repository archive , Bioc-2018-1
CRAN-2018-2: Changes in policy; packages should not give a check warning nor error. Bioc-2018-2: Moved to BiocManager to install packages.
CRAN-2019-1: More mentions to CRAN mirror security.

CRAN-2019-2: Updates in checklist for CRAN submissions, Bioc-2019-2
CRAN-2020-1: Many changes in CRAN policies. CRAN-2020-2: Many changes to CRAN policies. Bioc-2020-2: Announces the Technical and Community advisory boards (as well as the project-wide Code of Conduct).
CRAN-2021-1, Bioc-2021-1: Mentions conferences that will be virtual.
CRAN-2021-2: Shows an incomig path [See this friendly viewer, Bioc-2021-2: Mentions AnVIL and two online workshops to develop workflows.
CRAN-2022-1: List a change in CRAN policy and the CRAN Task View Initiative.
CRAN-2022-2: List some more repository policies. Bioc-2022-2: Lists infrastructure updates (and its funding), changes in the core team and new initiatives.
CRAN-2022-3, Bioc-2022-3
CRAN-2022-4, Bioc-2022-4: default branch renaming, partnership with Outreachy and blog are featured. Several working groups provide updates
CRAN-2023-1

In addition, several articles and blog posts have appeared. From those I found it is worth mentioning the following:

And my own posts:

Reasons CRAN packages are archived
CRAN files part 1
CRAN files part 2
CRAN maintained packages
CRAN review (and the talk at useRs 2021)
Bioconductor review
rOpenSci reviews
Bioconductor reviews
The article “Aspects of the Social Organization and Trajectory of the R Project”, from the R Journal 2009, also has a section about CRAN, noting that it “is challenged by its own success”.

Characteristics

The predominance of CRAN and its role as primary and default R repository has lead to some special treatment of the repository.

CRAN checks are in the R source code itself. While other repositories have their own checks in different tools. In addition, the CRAN environmental variables used are documented in the R-internals (they are more or less accessible in the svn repository too).

Others who know more have stated the benefits of CRAN too: This text is copied from Henrik Bengstsson in Bioconductor Slack:

FOREVER ARCHIVE:

The first one is that it publishes packages and versions of them until the end of time. When a package has been published on CRAN, it takes a lot for it to be removed from there. I don’t know if it ever happened, but I can imagine a package can be fully removed if it was illegally published in the first place (e.g. copyright, illegal content, ...) or malicious.

INSTALLATION SERVICE:

Then CRAN also provides a R package repository service for installing packages on CRAN using built-in R functions. The set of packages in the package repo is a subset of all packages on CRAN. The CRAN package repo makes a promise that all packages listed in PACKAGES can be installed. If they cannot make that promise, they’ll archive the package (=remove it from PACKAGES). I should also say, install.packages(url) can be used to install from the set of packages that are archived. Technically, old package versions are always archived.

CHECK SERVICE:

The content of the R package repository is guided by the CRAN package checks that run on R-oldrel, R-release, and R-devel across multiple platforms. The minimal requirement is that no package should remain in the package repository if the checks detects ERRORs (and those errors are not due to recently introduced bugs in R-devel). WARNINGs can also cause a package to be archived, but that process often takes longer. AFAIK, NOTEs are not a cause for a package being archived (but I could be wrong). The CRAN incoming checks, which you have to pass when you submit a new package, or an updated version, will make sure that the published package pass with all OKs. (It’s possible to argue for NOTEs being false positives, or for them not to be fixed, but that requires a manual approval by the CRAN Team).

I think there are many more resources discussing R repositories. If you know more I’ll be happy to update this post.

I think before I drag too much on the steps I’ll post this and collect some more articles I might have missed.

Last, Uwe Liegges presented about CRAN in useR!2017, thanks Tim Taylor for sharing it. In this video there is an explanation of why the solaris OS was used.

It has come to my attention that there is an article, by G. Brooke Anderson and Dirk Eddelbuette, about the R package repositories structure (among other things): Hosting Data Packages via drat: A Case Study with Hurricane Exposure Data

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       Ubuntu 22.04.3 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2024-01-15
##  pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date (UTC) lib source
##  blogdown      1.18    2023-06-19 [1] CRAN (R 4.3.1)
##  bookdown      0.37    2023-12-01 [1] CRAN (R 4.3.1)
##  bslib         0.6.1   2023-11-28 [1] CRAN (R 4.3.1)
##  cachem        1.0.8   2023-05-01 [1] CRAN (R 4.3.1)
##  cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.1)
##  digest        0.6.33  2023-07-07 [1] CRAN (R 4.3.1)
##  evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.2)
##  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.1)
##  htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.2)
##  jquerylib     0.1.4   2021-04-26 [1] CRAN (R 4.3.1)
##  jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.3.1)
##  knitr         1.45    2023-10-30 [1] CRAN (R 4.3.2)
##  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.2)
##  R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.1)
##  rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.1)
##  rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.1)
##  rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
##  sass          0.4.8   2023-12-06 [1] CRAN (R 4.3.1)
##  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.1)
##  xfun          0.41    2023-11-01 [1] CRAN (R 4.3.2)
##  yaml          2.3.8   2023-12-11 [1] CRAN (R 4.3.1)
## 
##  [1] /home/lluis/bin/R/4.3.1
##  [2] /opt/R/4.3.1/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

In version 3.1.2 Omegahat didn’t provide Windows binaries and in 4.1 from the default repositories (See 4.1 in NEWS(.4)).↩︎
This lead to the need of having a special function to install packages from Bioconductor. Initially a function biocLite and later with the BiocManager package.↩︎
NEWS in 2.15 section ↩︎

BaseSet 0.9.0

Wed, 23 Aug 2023 00:00:00 +0000

I’m excited to provide a new release of BaseSet, the package implementing a a class and methods to work with (fuzzy) sets.

This release was focused on making it easier to work with it.

From the beginning it was engineered towards the tidyverse and this time I focused on general R methods like $, [, c:

New methods

First we can create a TidySet or TS for short:

library("BaseSet", warn.conflicts = FALSE)
packageVersion("BaseSet")
## [1] '0.9.0'
l <- list(A = "1",
     B = c("1", "2"),
     C = c("2", "3", "4"),
     D = c("1", "2", "3", "4")
)
TS <- tidySet(l)

Up till now there was no compatibility with the base R methods but there was with the tidyverse.

TSa <- TS[["A"]]
TSb <- TS[["B"]]

Maybe this doesn’t look much but previously it wasn’t possible to subset the class. Initially I thought that working with a single class per session would be enough. Later I realized that maybe people would have good reasons to split or combine multiple objects:

TSab <- c(TSa, TSb)
TSab
##   elements sets fuzzy
## 1        1    A     1
## 2        1    B     1
## 3        2    B     1

Note that subsetting by sets does not produce the same object as elements are kept:

dim(TSab)
##  Elements Relations      Sets 
##         2         3         2
dim(TS[1:2, "sets"])
##  Elements Relations      Sets 
##         4         3         2

You’ll need to drop the elements:

dim(droplevels(TS[1:2, "sets"]))
##  Elements Relations      Sets 
##         2         3         2

We can include more information like this:

TSab[1:2, "relations", "type"] <- c("new", "addition")
TSab[1:2, "sets", "origin"] <- c("fake", "real")
TSab
##   elements sets fuzzy     type origin
## 1        1    A     1      new   fake
## 2        1    B     1 addition   real
## 3        2    B     1     <NA>   real

With this release is easier to access the columns of the TidySet:

TSab$type
## [1] "new"      "addition" NA
TSab$origin
## [1] "fake" "real"
TS$sets
##  [1] "A" "B" "B" "C" "C" "C" "D" "D" "D" "D"

If you pay attention you’ll realize that it will look at the minimum information required. But if the column is present in the relations and elements or sets slots it will pick the first.

You can use:

TS[, "sets", "new"] <- "a"
TS[, "sets", "new"]

I recommend reading carefully the help page of ?`extract-TidySet` and make some tests based on the examples. I might have created some bugs or friction points with the extraction operations, let me know and I’ll address them (That’s the reason why I kept it below a 1.0 release).

More usable

Another usability addition to the class is the possibility to autocomplete.

Now if you tab TS$ty and press TAB it should complete to TS$type because there is a column called type. This will make it easier to use the $.

With this release, we can now check the number of sets and the number of relations each set has:

length(TS)
## [1] 4
lengths(TS)
## A B C D 
## 1 2 3 4

New function

The new function union_closed checks if the combinations of sets produce already existing sets.

union_closed(TS, sets = c("A", "B", "C"))
## [1] FALSE
union_closed(TS)
## [1] TRUE

Next steps

I hope this makes it even easier to work with the class. Combine different objects, and manipulate it more intuitively.

While creating this document I realized it has some friction points.
In next release it will be possible to:

Subset the object by element or set name, if only querying elements and sets slots. For example TS[c("3", "4"), "elements", "NEWS"] <- TRUE
Use names and dimnames to discover which data is in the object.
Some bug fixes about these methods.

Enjoy!

I would also apreciate to hear some feedback about how you are using the package. It will help me to direct the development/maintenance of the package wherever it is more useful.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       Ubuntu 22.04.3 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2023-12-18
##  pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date (UTC) lib source
##  BaseSet     * 0.9.0   2023-08-23 [1] local
##  blogdown      1.18    2023-06-19 [1] CRAN (R 4.3.1)
##  bookdown      0.37    2023-12-01 [1] CRAN (R 4.3.1)
##  bslib         0.6.1   2023-11-28 [1] CRAN (R 4.3.1)
##  cachem        1.0.8   2023-05-01 [1] CRAN (R 4.3.1)
##  cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.1)
##  digest        0.6.33  2023-07-07 [1] CRAN (R 4.3.1)
##  dplyr         1.1.4   2023-11-17 [1] CRAN (R 4.3.1)
##  evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.2)
##  fansi         1.0.6   2023-12-08 [1] CRAN (R 4.3.1)
##  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.1)
##  generics      0.1.3   2022-07-05 [1] CRAN (R 4.3.1)
##  glue          1.6.2   2022-02-24 [1] CRAN (R 4.3.1)
##  htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.2)
##  jquerylib     0.1.4   2021-04-26 [1] CRAN (R 4.3.1)
##  jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.3.1)
##  knitr         1.45    2023-10-30 [1] CRAN (R 4.3.2)
##  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.2)
##  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.1)
##  pillar        1.9.0   2023-03-22 [1] CRAN (R 4.3.1)
##  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.3.1)
##  R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.1)
##  rlang         1.1.2   2023-11-04 [1] CRAN (R 4.3.1)
##  rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.1)
##  rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
##  sass          0.4.8   2023-12-06 [1] CRAN (R 4.3.1)
##  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.1)
##  tibble        3.2.1   2023-03-20 [1] CRAN (R 4.3.1)
##  tidyselect    1.2.0   2022-10-10 [1] CRAN (R 4.3.1)
##  utf8          1.2.4   2023-10-22 [1] CRAN (R 4.3.2)
##  vctrs         0.6.5   2023-12-01 [1] CRAN (R 4.3.1)
##  xfun          0.41    2023-11-01 [1] CRAN (R 4.3.2)
##  yaml          2.3.8   2023-12-11 [1] CRAN (R 4.3.1)
## 
##  [1] /home/lluis/bin/R/4.3.1
##  [2] /opt/R/4.3.1/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

experDesign: follow up

Sun, 09 Apr 2023 00:00:00 +0000

I am happy to announce a new release of experDesign. Install it from CRAN with:

install.packages("experDesign")
library("experDesign")

This new release has focused in more tricky aspects when designing an experiment:

Checking the samples of your experiment.
How to continue stratifying your conditions after some initial batch.

These functions should be used before carrying out anything once you have your samples collected. You can use these functions and make an informed decision of what might happen with your experiment.

Checking your samples

The new function check_data() will warn you if it finds some known issues with your data.

library("experDesign")
library("MASS")

If we take the survey dataset from the MASS package we can see that it has some issues:

data(survey, package = "MASS")
check_data(survey)
## Warning: Two categorical variables don't have all combinations.
## Warning: Some values are missing.
## Warning: There is a combination of categories with no replicates; i.e. just one
## sample.
## [1] FALSE

While if we fabricate our own dataset we might realize we have a problem

rdata <- expand.grid(sex = c("M", "F"), class = c("lower", "median", "high"))
stopifnot("Same samples/rows as combinations of classes" = nrow(rdata) == 2*3)
check_data(rdata)
## Warning: There is a combination of categories with no replicates; i.e. just one
## sample.
## [1] FALSE
# We create some new samples with the same conditions
rdata2 <- rbind(rdata, rdata)
check_data(rdata2)
## [1] TRUE

One might decide to go ahead with what is available or use only some of those samples or wait to collect more samples for the experiment

Follow up

Imagine you have 100 samples that you distribute in 4 batches of 25 samples each. Later, you collect 80 more samples to analyze. You want these new samples to be analyzed together with those previous 100 samples. Will it be possible? How should you distribute your new samples in groups of 25?

Using the same dataset from MASS imagine if we first collected 118 observations and later 119 more:

survey1 <- survey[1:118, ]
survey2 <- survey[119:nrow(survey), ]
# Using low number of iterations to speed the process 
# you should even use higher number than the default
fu <- follow_up(survey1, survey2, size_subset = 50, iterations = 10)
## Warning: There are some problems with the data.
## Warning: There are some problems with the new samples and the batches.
## Warning: There are some problems with the new data.
## Warning: There are some problems with the old data.

Following the previous new function it reports if there are problems with the observations. One can check each collection with check_data to know more about the problems found.

If you have already performed the experiment on your observations you can also check the distribution:

# Create the first batch
variables <- c("Sex", "Smoke", "Age")
survey1 <- survey1[, variables]
index1 <- design(survey1, size_subset = 50, iterations = 10)
## Warning: There might be some problems with the data use check_data().
r_survey <- inspect(index1, survey1)
# Create the second batch with "new" students
survey2 <- survey2[, variables]
survey2$batch <- NA
# Prepare the follow up
all_classroom <- rbind(r_survey, survey2)
fu2 <- follow_up2(all_classroom, size_subset = 50, iterations = 10)
## Warning: There are some problems with the data.
## Warning: There are some problems with the new samples and the batches.
## Warning: There are some problems with the new data.
## Warning: There are some problems with the old data.
tail(fu2)
## [1] "NewSubset2" "NewSubset2" "NewSubset2" "NewSubset2" "NewSubset2"
## [6] "NewSubset3"

Using this function will help to decide which new observations go to which new batches.

Closing remarks

The famous quote from Fisher goes:

“To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.”

This emphasizes the importance of involving a statistician early on in the experimental design process.
Unfortunately, in some cases, it may be too late to involve a statistician in the experimental design process or the reality of unforeseen circumstances messed the design of your carefully planned experiment.

My aim with this package is to provide practical tools for statisticians, bioinformaticians, and anyone who works with data. These tools are designed to be easy to use and can be used to analyze data in a variety of contexts. Let me know if it is helpful in your case.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.2.2 (2022-10-31)
##  os       Ubuntu 22.04.2 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language en_US
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2023-04-09
##  pandoc   2.19.2 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version  date (UTC) lib source
##  blogdown      1.16     2022-12-13 [1] CRAN (R 4.2.2)
##  bookdown      0.33     2023-03-06 [1] CRAN (R 4.2.2)
##  bslib         0.4.2    2022-12-16 [1] CRAN (R 4.2.2)
##  cachem        1.0.7    2023-02-24 [1] CRAN (R 4.2.2)
##  cli           3.6.1    2023-03-23 [1] CRAN (R 4.2.2)
##  digest        0.6.31   2022-12-11 [1] CRAN (R 4.2.2)
##  evaluate      0.20     2023-01-17 [1] CRAN (R 4.2.2)
##  experDesign * 0.2.0    2023-04-05 [1] CRAN (R 4.2.2)
##  fastmap       1.1.1    2023-02-24 [1] CRAN (R 4.2.2)
##  htmltools     0.5.4    2022-12-07 [1] CRAN (R 4.2.2)
##  jquerylib     0.1.4    2021-04-26 [1] CRAN (R 4.2.2)
##  jsonlite      1.8.4    2022-12-06 [1] CRAN (R 4.2.2)
##  knitr         1.42     2023-01-25 [1] CRAN (R 4.2.2)
##  MASS        * 7.3-58.1 2022-08-03 [2] CRAN (R 4.2.2)
##  R6            2.5.1    2021-08-19 [1] CRAN (R 4.2.2)
##  rlang         1.1.0    2023-03-14 [1] CRAN (R 4.2.2)
##  rmarkdown     2.20     2023-01-19 [1] CRAN (R 4.2.2)
##  rstudioapi    0.14     2022-08-22 [1] CRAN (R 4.2.2)
##  sass          0.4.5    2023-01-24 [1] CRAN (R 4.2.2)
##  sessioninfo   1.2.2    2021-12-06 [1] CRAN (R 4.2.2)
##  xfun          0.37     2023-01-31 [1] CRAN (R 4.2.2)
##  yaml          2.3.7    2023-01-23 [1] CRAN (R 4.2.2)
## 
##  [1] /home/lluis/bin/R/4.2.2
##  [2] /opt/R/4.2.2/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Accessing REDCap from R

Wed, 08 Feb 2023 00:00:00 +0000

In this post, I want to summarize some of the packages to connect to REDCap. For those who don’t know, REDCap is a database designed for clinical usage, which allows easy data collection of patients’ responses by clinicians and interactions with the patients via surveys.

It has specific features such as scheduling surveys sent to patients, compatibility with tablets and mobile phones for data entry while visiting patients, grouping data in instruments (for repeating the same questions multiple times), multiple choice and check buttons, and different arms (like paths for patients). Most importantly is relatively easy to manage by clinical administrators.

In CRAN there are ~11 packages mentioning it at the time of writing it. The purpose of this post is to help decide which packages can be helpful in which situations. This post won’t be a deep analysis or comparison of capabilities, it describes some of the best and worse features of each package.

REDCapR

REDCapR is the official package to connect to the database. It allows you to read, write and filter the requests. It has some security-related functions.

REDCapTidieR

REDCapTidieR is a package that provides summaries of tables and helps with nested tibbles data by arm. It depends on REDCapR.

tidyREDCap

tidyREDCap is a package that simplifies the tables for instruments and choose-all or choose-one question types. It is easy to make tables and it depends on REDCapR. It requires the first and last columns to make instruments.

Screenshot of a design with several instruments in a single arm (from https://www.project-redcap.org/)

REDCapExporter

REDCapExporter is a package to build a data package from a database for redistribution. It does not depend on REDCapR.

redcapAPI

redcapAPI is a package for making data accessible and analysis-ready as quickly as possible with huge documentation in a wiki but has no vignette or examples and it does not depend on REDCapR.

REDCapDM

REDCapDM is a package that provides functions to read and manage REDCap data and identify missing or extreme values as well as transform the data provided by the API. It depends on REDCapR.

ReviewR

ReviewR is a package that creates a shiny website with data from the database to explore it. It uses the REDCapR to connect to your instance.

rccola

rccola is a package to provide a secure connection to the database but it doesn’t provide any handling of the data. It uses redcapAPI to connect to the database.

Other packages

Other packages mention REDCap:

nmadb: which implements its own connection procedure for a specific REDCap database of network meta-analyses.
distcomp: Allows to do computation on a distributed data also in REDCap.
cgmanalysis: which mentions that data produced is compatible with REDCap.

Conclusion

I’m sure that many packages briefly described here can do much more than what I understood from a glance at their documentation and DESCRIPTION.

Most packages provide some data for the examples (and probably tests), while others do not. This is a technical problem that might impact users if there are no examples in the functions.

REDCapR is used by most packages to access the database, but most of the packages focus on transforming the data provided by the API (or data exported) or the exported data. It highlights that the data exported is useful but that depending on the preferences of the users it needs to be transformed for easy usage.

Exploring CRAN's files: part 2

Thu, 28 Jul 2022 00:00:00 +0000

Introduction

In the first post of the series we briefly explored packages available on CRAN. Now I’ll focus on history of the packages and its size using the following files:

packages <- tools::CRAN_package_db()
current <- tools:::CRAN_current_db()
archive <- tools:::CRAN_archive_db()

In this part we will use two files: The current and the archive, let’s see why.

current file

The current database has has the package size, dates of modification, which I assume is date added to CRAN and user name of who last modified it. This is the same information returned by file.info

current[1, 1:10]
##     size isdir mode               mtime               ctime               atime
## A3 42810 FALSE  664 2015-08-16 23:05:54 2022-09-03 12:02:27 2022-09-03 14:00:19
##     uid  gid  uname    grname
## A3 1001 1001 hornik cranadmin

archive file

The archive database returns the same information, but as you might guess by the name it doesn’t provide information about current packages but for packages in the archive and no longer available by default.

archive[[1]]
##                     size isdir mode               mtime               ctime
## A3/A3_0.9.1.tar.gz 45252 FALSE  664 2013-02-07 10:00:29 2022-08-22 18:14:53
## A3/A3_0.9.2.tar.gz 45907 FALSE  664 2013-03-26 19:58:40 2022-08-22 18:14:53
##                                  atime  uid  gid  uname    grname
## A3/A3_0.9.1.tar.gz 2022-08-22 17:39:50 1001 1001 hornik cranadmin
## A3/A3_0.9.2.tar.gz 2022-08-22 17:39:50 1010 1001 ligges cranadmin

The date matches that available on the web’s old sources, so we can be confident of it’s meaning.

CRAN history

As we have seen there are some files about the archives of CRAN. These include information about date of modification (moving/editing) and user who did it and of course name and sometimes version of the package. These archives are the great treasure of CRAN because they help to make reproducible long time ago run experiments or analysis.

Note that I’m not totally sure that this archive contains the full record of packages, some initial packages might be missing. I’m also aware of some packages removed by CRAN which do not longer appear on this records.

Nevertheless, this should provide an accurate picture of packages available through time. Also as there is no information when a package is archived (here, there is on PACKAGES.in) so I might overestimate the packages available at any given moment.

Remember the plot about acceptance of packages on CRAN? That plot only looked at current packages available, let’s check it with all the archive:

Figure 1: Packages on CRAN archive by their addition to it. There are over 125000 archives on CRAN.

All these packages come from packages with few releases and packages with many releases. If we look at which packages had the most releases:

package	Releases
spatstat	206
Matrix	204
mgcv	162
RcppArmadillo	150
rgdal	146
nlme	143
caret	139
spdep	139
lattice	137
plotrix	131
sp	128
XML	126
Rcmdr	123
lme4	122
gstat	121
arm	119
foreign	117
party	117
maptools	113
raster	108

Surprisingly there are packages with more than 200 versions on CRAN!

Figure 2: Releases distirbution. Packages and number of releases

Most packages have 1 release, usually packages have 3, but the mean is around 6.

Given all this different versions of packages how big are all the packages on CRAN?

CRAN size

Have you ever wondered how big is CRAN? According to the memory size of the source packages all CRAN source packages are approximately 96.8 Gb.

This doesn’t include binaries for multiple architectures and OS. The package size might indicate whether the package has considerable amount of data.

Looking back to the size of the packages along time we can see this pattern:

Figure 3: Package and their median size. Archived packages have become bigger since 2014. Packages on CRAN have been getting bigger since 2017.

Packages available on CRAN are smaller than those no longer on CRAN. But versions of packages on CRAN that got archived are usually bigger than current versions. Packages no longer on CRAN are usually bigger. Median size of packages is increasing (quickly).

Figure 4: Size of package with releases. Package are usually small but seem to gain weight when updating.

Typically packages increase their size with each new release up to when they reach 50 releases. For higher releases this plot depends on very few packages and might not be representative.

Figure 5: Size of package with releases by availability. Packages no longer in CRAN are usually smaller than those in it. The continous black line is CRAN’s current threshold, while the discontinous black line is current median size.

Here we can appreciate better how packages tend to be below the CRAN threshold. There isn’t much of a difference between packages available on CRAN and those archived.

If we look at the size of package of the first release over time we’ll see a representative view:

Figure 6: Size of the first release by time. Package size increases with time with a peak around 2010 and increasing again since 2014 but still hasn’t surprased the previous record.

Package size tends to increase except for the brief period 2010-2014. Currently it increases less than before that period but is close to its maximum.

Conclusions

Most packages are not updated too much, between 1 and 3 times. But there are packages that are updated quite a lot, this might mean they are data packages and not software packages or that they have frequent minor and major updates.
Most current packages have smaller size than those archived. Packages no longer available usually had bigger size than those packages still on CRAN.
Surprisingly packages increase their size a lot till the 25 release. But also with time except for a period in 2010 and 2014. This decreasing period might be due to a change in CRAN policy.

Future parts

On future posts I’ll explore:

patterns accepting packages and updates in packages.
the relation between dependencies, initial release and updates.
who handled the packages.

Exploring CRAN's files: part 1

Sat, 23 Jul 2022 00:00:00 +0000

Introduction

There are many great things in base R, one of them is the tools package. This package has the functions that are used to build, check and create packages, documentation and manuals.

As I wanted to know how CRAN works and its changes I was looking into the source code of tools. I found some internal functions that access freely available files with information about CRAN packages. These private functions are at the CRANtools.R file.

packages <- tools::CRAN_package_db()
# current <- tools:::CRAN_current_db()
# archive <- tools:::CRAN_archive_db()
# issues <- tools::CRAN_check_issues()
# alias <- tools:::CRAN_aliases_db()
# rdxrefs <- tools:::CRAN_rdxrefs_db()

As I was not sure of the information on these files I asked on R-devel but I did not receive an answer. They seem to be quite obscure and as private functions might be removed without notice and shouldn’t be used in any dependency. However, as the files contain information about CRAN they might provide interesting clues about the history of CRAN and how it is operated.

On this post I will focus on the first file. I’ll explore a couple of fields and in future posts I will use the other files to explore more about CRAN history.

packages file

First of all a very brief exploration of what is in this file:

##    Package Version Priority                        Depends
## 1       A3   1.0.0     <NA> R (>= 2.15.0), xtable, pbapply
## 2 AATtools   0.0.1     <NA>                   R (>= 3.6.0)
## 3   ABACUS   1.0.0     <NA>                   R (>= 3.1.0)
##                                 Imports LinkingTo
## 1                                  <NA>      <NA>
## 2  magrittr, dplyr, doParallel, foreach      <NA>
## 3 ggplot2 (>= 3.1.0), shiny (>= 1.3.1),      <NA>
##                               Suggests Enhances    License License_is_FOSS
## 1                  randomForest, e1071     <NA> GPL (>= 2)            <NA>
## 2                                 <NA>     <NA>      GPL-3            <NA>
## 3 rmarkdown (>= 1.13), knitr (>= 1.22)     <NA>      GPL-3            <NA>

Packages has similar information as available.packages() but with many more columns with published date, reverse dependencies, X-CRAN-Comment, who packaged it… Also note that all this packages are not filtered to match R version, OS_type, subarch and there are almost duplicates (I learned about this filtering while reading the great documentation of available.packages() and also finding some mentions online).

As we have data from several years I’ll sometimes show the release dates of different R versions to provide some context. Without further delay let’s explore the data!

Published packages

CRAN started some time ago (in 1997) but it hasn’t remained frozen. The package archive (the A in CRAN) has been updating since then. For instance the current packages do not include packages that were removed, archived or those replaced by updates.

First packages are submitted to CRAN and once accepted they are published. As accepted and published usually are almost instantaneous I might use them as synonyms. Looking at the current available packages and their publication date, we can see the following:

Figure 1: Packages accepted on CRAN by the publication date.

The oldest package added was in 2010. This means a package without issues, dependencies changes, bugs detected by the automatic checks since 12 years!

The daily rate of acceptance has increased from less than 10 a day till 2020 to more than 30 this year 2022. If we summarize that information for month we see the same, but the little bump in 2020 disappears but we see other patterns:

Figure 2: Monthly packages published to CRAN. Some monthly variance is observed.

Instead of just one bump we see some waves with less packages on CRAN accepted late in the year and an increase of packages the first months of the year.

If we look at the accumulated packages on CRAN we see an exponential growth:

Figure 3: Acumulation of packages. Most of the packages have been published in the last 2 years.

In fact, most packages currently on CRAN where added since March 2021 than all the previous years.

Figure 4: Percentage of current packages on CRAN according to their date of publication. Most of them were published/updated on the last year and a half.

This is a good time to remind that the date being used is the date of publication of this version of the packages. Many had previous versions on CRAN:

First release	Packages
No	14,294
Yes	4,113

Processing time

Previously I found that CRAN submissions present some key differences between new packages and already published packages which impact how long do they need to wait to be published on CRAN. With the existing data we can compare how fast is the process by comparing the published date with the build date.

The build date is added to the tar.gz file automatically when the developer builds the package via R CMD build. However, the published date is set by CRAN once the packages are accepted on CRAN.

To visualize the differences I will also compare if there is some difference with new packages and those that were already on CRAN:

Figure 5: Histogram of time difference between building and publishing a package. Color indicates if the package is new to CRAN or not. Most of the published packages take more or less the same time regardless of if it is the first time or not.

There doesn’t seem to be much difference between date of building and date of publication according to if it is the first release or not. The precision is just a day and this is usually a fast process well below 50 days. Few packages exceed spend so much after build before publication and they are too few to be noticeable at this scale. Since 2016/05/02 there is a check that raises an issue if the build is older than a month.

Note that one might need to build multiple times the package before it is accepted. Packages published for the first time on CRAN might have been submitted previously, but when they finally built and pass the checks and manual review they are handled as fast as packages already on CRAN.

However, this time between build and acceptance might have changed with time:

Figure 6: Processing time between building the package and being published by date. There is a high difference between new packages and old ones. New packages usually take more time while existing packages take less than a day currently.

We clearly see a difference in processing time for those packages already on CRAN and those that are not. Keep in mind that for the few packages from before 2016 the estimation might not be accurate. At the same time this is consistent with the manual review process (For more information see my previous post about the review process of CRAN or my talk at the useR2021). It also means that there is a huge variation of time about how packages are handled. However this seems to be reducing: while in 2010 it took around 2 weeks, nowadays it takes less than a week and getting closer to a 1 day of median time between a package being built and appearing on CRAN that takes for existing packages.

This difference might be explainable due to experience: authors and maintainers whose package(s) are already in CRAN know better how to submit a new version without problems the checks.

It could also be that new packages need more time from the CRAN team. In 2020 we see it took longer than in previous years for packages to be added on CRAN. Maybe the increase in the processing time in 2020 was due the huge volume of submissions CRAN received or more checks on the developer side before submitting it to CRAN.

Both explanations are not mutually exclusive.

More packages published the same day mean more processing time? It doesn’t look like it.

Figure 7: Packages accepted the same day and processing time.New packages are accepted sooner than packages on CRAN respect to the builddate.

Surprisingly, we see a lot of variation on the delay of packages already accepted on CRAN. In addition, the more new packages accepted the same day, the less delay there is. I think this just means that when reviewers work on the submission queue several packages might be approved.

This might also mean packages have already been built several times before finally being accepted and now the errors, warnings and notes have been solved. Last, this could indicate that developers with their package already on CRAN wait a bit between building and submitting the package as the developer might be taking some time to double check before submission (dependencies, on several machines, other?) or a time zone difference (submitting in the noon of a region but at the reviewers night).

Conclusion

There are packages that for 12 years have been working without problems despite the several major changes in R (See figure 1). This speaks volumes of the packages’ quality, and the backward compatibility that the R core aims and CRAN checks.
CRAN accepts an incredible amount of packages daily and monthly. The system and the team are doing an incredible work mostly on their free time (See figure 2). Many thanks!
Accepted packages are handled very fast, in less than a week usually (See figure 7). But it is not possible to distinguish alone time in the submission system and time on the developer computer.

Future parts

We’ve explored a snapshot of current packages and a brief window of all the history of CRAN. There is much more that can be done with all the other files.

On future posts I’ll explore:

patterns accepting packages and updates in packages.
who handled the packages.
Size of packages.
the relation between dependencies, initial release and updates.

Other suggestions?

Edit: Many thanks to Maëlle Salmon and Dirk Eddelbuettel for their feedback on an initial version of this series of posts.

Reproducibility

## - Session info -------------------------------------------------------------------------------------------------------
##  setting  value
##  version  R version 4.2.1 (2022-06-23)
##  os       Ubuntu 20.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  C
##  ctype    C
##  tz       Europe/Madrid
##  date     2022-07-23
##  pandoc   2.18 @ /usr/lib/rstudio/bin/quarto/bin/tools/ (via rmarkdown)
## 
## - Packages -----------------------------------------------------------------------------------------------------------
##  package      * version    date (UTC) lib source
##  assertthat     0.2.1      2019-03-21 [2] RSPM (R 4.2.0)
##  base64enc      0.1-3      2015-07-28 [2] CRAN (R 4.0.0)
##  blogdown       1.10       2022-05-10 [2] RSPM (R 4.2.0)
##  bookdown       0.27       2022-06-14 [2] RSPM (R 4.2.0)
##  bslib          0.4.0      2022-07-16 [2] RSPM (R 4.2.0)
##  cachem         1.0.6      2021-08-19 [2] RSPM (R 4.2.0)
##  cli            3.3.0      2022-04-25 [2] RSPM (R 4.2.0)
##  codetools      0.2-18     2020-11-04 [2] RSPM (R 4.2.0)
##  colorspace     2.0-3      2022-02-21 [2] RSPM (R 4.2.0)
##  crayon         1.5.1      2022-03-26 [2] RSPM (R 4.2.0)
##  curl           4.3.2      2021-06-23 [2] RSPM (R 4.2.0)
##  data.table     1.14.2     2021-09-27 [2] RSPM (R 4.2.0)
##  DBI            1.1.3      2022-06-18 [2] RSPM (R 4.2.0)
##  digest         0.6.29     2021-12-01 [2] RSPM (R 4.2.0)
##  dplyr        * 1.0.9      2022-04-28 [2] RSPM (R 4.2.0)
##  ellipsis       0.3.2      2021-04-29 [2] RSPM (R 4.2.0)
##  evaluate       0.15       2022-02-18 [2] RSPM (R 4.2.0)
##  fansi          1.0.3      2022-03-24 [2] RSPM (R 4.2.0)
##  farver         2.1.1      2022-07-06 [2] RSPM (R 4.2.0)
##  fastmap        1.1.0      2021-01-25 [2] RSPM (R 4.2.0)
##  flextable    * 0.7.2      2022-06-12 [2] RSPM (R 4.2.0)
##  forcats      * 0.5.1      2021-01-27 [2] RSPM (R 4.2.0)
##  gdtools        0.2.4      2022-02-14 [2] RSPM (R 4.2.0)
##  generics       0.1.3      2022-07-05 [2] RSPM (R 4.2.0)
##  geomtextpath * 0.1.0      2022-01-24 [2] CRAN (R 4.2.1)
##  ggplot2      * 3.3.6.9000 2022-06-29 [2] Github (tidyverse/ggplot2@7571122)
##  ggrepel      * 0.9.1      2021-01-15 [2] RSPM (R 4.2.0)
##  glue           1.6.2      2022-02-24 [2] RSPM (R 4.2.0)
##  gtable         0.3.0      2019-03-25 [2] CRAN (R 4.0.0)
##  highr          0.9        2021-04-16 [2] RSPM (R 4.2.0)
##  htmltools      0.5.3      2022-07-18 [2] RSPM (R 4.2.0)
##  jquerylib      0.1.4      2021-04-26 [2] RSPM (R 4.2.0)
##  jsonlite       1.8.0      2022-02-22 [2] RSPM (R 4.2.0)
##  knitr          1.39       2022-04-26 [2] RSPM (R 4.2.0)
##  labeling       0.4.2      2020-10-20 [2] RSPM (R 4.2.0)
##  lattice        0.20-45    2021-09-22 [3] CRAN (R 4.2.0)
##  lifecycle      1.0.1      2021-09-24 [2] RSPM (R 4.2.0)
##  lubridate    * 1.8.0      2021-10-07 [2] RSPM (R 4.2.0)
##  magrittr       2.0.3      2022-03-30 [2] RSPM (R 4.2.0)
##  Matrix         1.4-1      2022-03-23 [2] RSPM (R 4.2.0)
##  mgcv           1.8-40     2022-03-29 [2] RSPM (R 4.2.0)
##  munsell        0.5.0      2018-06-12 [2] RSPM (R 4.2.0)
##  nlme           3.1-158    2022-06-15 [2] RSPM (R 4.2.0)
##  officer        0.4.3      2022-06-12 [2] RSPM (R 4.2.0)
##  pillar         1.8.0      2022-07-18 [2] RSPM (R 4.2.0)
##  pkgconfig      2.0.3      2019-09-22 [2] RSPM (R 4.2.0)
##  purrr          0.3.4      2020-04-17 [2] RSPM (R 4.2.0)
##  R6             2.5.1      2021-08-19 [2] RSPM (R 4.2.0)
##  Rcpp           1.0.9      2022-07-08 [2] RSPM (R 4.2.0)
##  rlang          1.0.4      2022-07-12 [2] RSPM (R 4.2.0)
##  rmarkdown      2.14       2022-04-25 [2] RSPM (R 4.2.0)
##  rstudioapi     0.13       2020-11-12 [2] RSPM (R 4.2.0)
##  rversions    * 2.1.1      2021-05-31 [2] RSPM (R 4.2.0)
##  sass           0.4.2      2022-07-16 [2] RSPM (R 4.2.0)
##  scales         1.2.0      2022-04-13 [2] RSPM (R 4.2.0)
##  sessioninfo    1.2.2      2021-12-06 [2] RSPM (R 4.2.0)
##  stringi        1.7.8      2022-07-11 [2] RSPM (R 4.2.0)
##  stringr        1.4.0      2019-02-10 [2] RSPM (R 4.2.0)
##  systemfonts    1.0.4      2022-02-11 [2] RSPM (R 4.2.0)
##  textshaping    0.3.6      2021-10-13 [2] RSPM (R 4.2.0)
##  tibble         3.1.7      2022-05-03 [2] RSPM (R 4.2.0)
##  tidyr        * 1.2.0      2022-02-01 [2] RSPM (R 4.2.0)
##  tidyselect     1.1.2      2022-02-21 [2] RSPM (R 4.2.0)
##  utf8           1.2.2      2021-07-24 [2] RSPM (R 4.2.0)
##  uuid           1.1-0      2022-04-19 [2] RSPM (R 4.2.0)
##  vctrs          0.4.1      2022-04-13 [2] RSPM (R 4.2.0)
##  withr          2.5.0      2022-03-03 [2] RSPM (R 4.2.0)
##  xfun           0.31       2022-05-10 [2] RSPM (R 4.2.0)
##  xml2           1.3.3      2021-11-30 [2] RSPM (R 4.2.0)
##  yaml           2.3.5      2022-02-21 [2] RSPM (R 4.2.0)
##  zip            2.2.0      2021-05-31 [2] RSPM (R 4.2.0)
## 
##  [1] /home/lluis/bin/R/4.2.1
##  [2] /usr/lib/R/site-library
##  [3] /usr/lib/R/library
## 
## ----------------------------------------------------------------------------------------------------------------------

Upgrading rtweet to 1.0.2

Mon, 04 Jul 2022 00:00:00 +0000

In this post I will provide some examples of what has changed between rtweet 0.7.0 and rtweet 1.0.2. I hope both the changes and this guide will help all users. I highlight the most important and interesting changes in this blog post, and for a full list of changes you can consult it on the NEWS.

Big breaking changes

More consistent output

This is probably what will affect the most users. All functions that return data about tweets¹ now return the same columns.

For example if we search some tweets we’ll get the following columns:

> tweets <- search_tweets("weather")
> colnames(tweets)
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "full_text"                    
 [5] "truncated"                     "display_text_range"           
 [7] "entities"                      "metadata"                     
 [9] "source"                        "in_reply_to_status_id"        
[11] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[13] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[15] "geo"                           "coordinates"                  
[17] "place"                         "contributors"                 
[19] "is_quote_status"               "retweet_count"                
[21] "favorite_count"                "favorited"                    
[23] "retweeted"                     "lang"                         
[25] "quoted_status_id"              "quoted_status_id_str"         
[27] "quoted_status"                 "possibly_sensitive"           
[29] "retweeted_status"              "text"                         
[31] "favorited_by"                  "scopes"                       
[33] "display_text_width"            "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "query"                         "withheld_scope"               
[41] "withheld_copyright"            "withheld_in_countries"        
[43] "possibly_sensitive_appealable"

rtweet now minimizes the processing of tweets and only returns the same data as provided by the API while making it easier to handle by R. However, to preserve the nested nature of the data returned some fields are now nested inside other. For example, previously fields "bbpx_coords", "geo_coords", "coords_coords" were returned as separate columns, but they are now nested inside "place", "coordinates" or "geo" depending where they are provided. Some columns previously calculated by rtweet are now not returned, like "rtweet_favorite_count". At the same time it provides with new columns about each tweet like the "withheld_*" columns.

If you scanned through the columns you might have noticed that columns "user_id" and "screen_name" are no longer returned. This data is still returned by the API but it is now made available to the rtweet users via users_data():

> colnames(users_data(tweets))
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

This blog post should help you find the right data columns, but if you don’t find what you are looking for it might be nested inside a column.
Try using dplyr::glimpse() to explore the data and locate nested columns. For example the entities column (which is present in both tweets and users) have the following useful columns:

> names(tweets$entities[[1]])
[1] "hashtags"      "symbols"       "user_mentions" "urls"         
[5] "media"

Similarly if you look up a user via search_users() or lookup_users() you’ll get consistent data:

> users <- lookup_users(c("twitter", "rladiesglobal", "_R_Foundation"))
> colnames(users)
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

You can use tweets_data() to retrieve information about their latest tweet:

> colnames(tweets_data(users))
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "text"                         
 [5] "truncated"                     "entities"                     
 [7] "source"                        "in_reply_to_status_id"        
 [9] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[11] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[13] "geo"                           "coordinates"                  
[15] "place"                         "contributors"                 
[17] "is_quote_status"               "retweet_count"                
[19] "favorite_count"                "favorited"                    
[21] "retweeted"                     "lang"                         
[23] "retweeted_status"              "possibly_sensitive"           
[25] "quoted_status"                 "display_text_width"           
[27] "user"                          "full_text"                    
[29] "favorited_by"                  "scopes"                       
[31] "display_text_range"            "quoted_status_id"             
[33] "quoted_status_id_str"          "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "metadata"                      "query"                        
[41] "withheld_scope"                "withheld_copyright"           
[43] "withheld_in_countries"         "possibly_sensitive_appealable"

You can merge them via:

users_and_last_tweets <- cbind(users, id_str = tweets_data(users)[, "id_str"])

In the future (see below), with helper functions managing the output of rtweet will become easier.

Finally, get_followers() and get_friends() now return the same columns:

> colnames(get_followers("_R_Foundation"))
[1] "from_id" "to_id"  
> colnames(get_friends("_R_Foundation"))
[1] "from_id" "to_id"

This will make it easier to build networks of connections (although you might want to convert screen names to ids or vice versa).

More consistent interface

All paginated functions that don’t return tweets now use a consistent pagination interface (except the premium endpoints). They all store the “next cursor” in an rtweet_cursor attribute, which will be automatically retrieved when you use the cursor argument. This will make it easier to continue a query you started:

users <- get_followers("_R_Foundation")
users
     
# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

They support max_id and since_id to find earlier and later tweets respectively:

# Retrieve all the tweets made since the previous request
newer <- search_tweets("weather", since_id = tweets)
# Retrieve tweets made before the previous request
older <- search_tweets("weather", max_id = tweets)

If you want more tweets than it is allowed by the rate limits of the API, you can use retryonratelimit to wait as long as needed:

long <- search_tweets("weather", n = 1000, retryonratelimit = TRUE)

This will keep busy your terminal until the 1000 tweets are retrieved.

Saving data

An unexpected consequence of returning more data (now matching that returned by the API) is that it is harder to save it in a tabular format. For instance one tweet might have one media, mention two users and have three hashtags. There isn’t a simple way to save it in a single row uniformly for all tweets or it could lead to confusion.

This resulted in deprecating save_as_csv, read_twitter_csv and related functions because they don’t work with the new data structure and it won’t be possible to load the complete data from a csv. They will be removed in later versions.

Many users will benefit from saving to RDS (e.g., saveRDS() or readr::write_rds()), and those wanting to export to tabular format can simplify the data to include only that of interest before saving with generic R functions (e.g., write.csv() or readr::write_csv()).

Other breaking changes

Accessibility is important and for this reason if you tweet via post_tweet() and add an image, gif or video you’ll need to provide the media alternative text. Without media_alt_text it will not allow you to post.
tweet_shot() has been deprecated as it no longer works correctly. It might be possible to bring it back, but the code is complex and I do not understand enough to maintain it. If you’re interested in seeing this feature return, checkout the discussion about this issue and let me know if you have any suggestions.
rtweet also used to provide functions for data on emojis, langs and stopwordslangs. These are useful resources for text mining in general - not only in tweets - however they need to be updated to be helpful and would be better placed in other packages, for instance emojis is now on the bdpar package. Therefore they are no longer available in rtweet.
The functions like suggested_*() have been removed as they have been broken since 2019.

Easier authentication

An exciting part of this release has been a big rewrite of the authentication protocol. While it is compatible with previous rtweet authentication methods it has also some important new functions which make it easier to work with rtweet and the twitter API in different ways.

Different ways to authenticate

If you just want to test the package, use the default authentication auth_setup_default() that comes with rtweet. If you use it for one or two days you won’t notice any problem.

If you want to use the package for more than a couple of days, I recommend you set up your own token via rtweet_user(). It will open a window to authenticate via the authenticated account in your default browser. This authentication won’t allow you to do everything but it will avoid running out of requests and being rate-limited.

If you plan to make heavy use of the package, I recommend registering yourself as developer and using one of the following two mechanisms, depending on your plans:

Collect data and analyze: rtweet_app().
Set up a bot: rtweet_bot()

Find more information in the Authentication with rtweet vignette.

Storing credentials

Previously rtweet saved each token created, but now non-default tokens are only saved if you ask. You can save them manually via auth_save(token, "my_app"). Bonus, if you name your token as default (auth_save(token, "default")) it will be used automatically upon loading the library.

Further, tokens are now saved in the location output by tools::R_user_dir("rtweet", "config"), rather than in your home directory. If you have previous tokens saved or problems identifying which token is which use auth_sitrep(). This will provides clues to which tokens might be duplicated or misconfigured but it won’t check if they work. It will also automatically move your tokens to the new path.

To check which credentials you have stored use auth_list() and load them via auth_as("my_app"). All the rtweet functions will use the latest token loaded with auth_as (unless you manually specify one when calling it). If you are not sure which token you are using you can use auth_get() it will return the token in use, list them or ask you to authenticate.

Other changes of note

This is a list of other changes that aren’t too big or are not breaking changes but are worthy enough of a mention:

Iteration and continuation of requests

Using cursors, pagination or waiting until you can make more queries is now easier. For example you can continue previous requests via:

users <- get_followers("_R_Foundation")
users

# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

Additions

There is now a function to find a thread of a user. You can start from any tweet and it will find all the tweets of the thread: tweet_threading("1461776330584956929").

There is a lot of interest in downloading and keeping track of interactions on Twitter. The amount of interest is big enough that Twitter is releasing a new API to provide more information of this nature.

Future

Twitter API v2 is being released and soon it will replace API v1. rtweet up to now, including this release, uses API v1 so it will need to adapt to the new endpoints and new data returned.

First will be the streaming endpoints in November, so expect more (breaking?) changes around those dates if not earlier.

I would also like to make it easier for users, dependencies and the package itself to handle the outputs. To this regard I would like to provide some classes to handle the different type of objects it returns.

This will help avoid some of the current shortcomings. Specifically I would like to provide functions to make it easier to reply to previous tweets, extract nested data, and subset tweets and the accompanying user information.

Conclusions

While I made many breaking changes I hope these changes will smooth future development and help both users and maintainers.

Feel free to ask on the rOpenSci community if you have questions about the transition or find something amiss. Please let me know! It will help me prioritize which endpoints are more relevant to the community. (And yes, the academic archive endpoint is on the radar.)

It is also possible that I overlooked something and I thought the code is working when it isn’t. For example, after several months of changing the way the API is parsed, several users found it wasn’t handling some elements. Let me know of such or similar cases and I’ll try to fix it.

In case you find a bug, check the open issues and if it has not already been reported, open an issue on GitHub. Don’t forget to make a reprex and if possible provide the id of the tweets you are having trouble with. Unfortunately it has happened that when I came to look at a bug I couldn’t reproduce it as I wasn’t able to find the tweet which caused the error.

This release includes contributions from Hadely Wicham, Bob Rudis, Alex Hayes, Simon Heß, Diego Hernán, Michael Chirico, Jonathan Sidi, Jon Harmon, Andrew Fraser and many other that reported bugs or provided feedback. Many thanks all for using it, your interest to keep it working and improving rtweet for all.

Finally, you can read the whole NEWS online and the examples.

Happy tweeting!

Acknowledgements

This is a repost of the entry for rOpenSci. The post was edited and improved by Yanina Bellini Saibene and Steffi LaZerte, the community manager and assistant. Many thanks

Specifically these: get_favorites(), get_favorites_user(), get_mentions(), get_my_timeline(), get_retweets(), get_timeline(), get_timeline_user(), lists_statuses(), lookup_statuses(), lookup_tweets(), search_30day(), search_fullarchive(), search_tweets(), tweet_shot() and tweet_threading().↩︎

Writing a thesis with bookdown

Mon, 09 May 2022 00:00:00 +0000

On this post I am documenting the experiences I had writing my PhD thesis with bookdown. I made the thesis in web and pdf format (and epub) to make more available the thesis. Most of the experiences and advise I’ll share here are based on my experiences to improve the pdf format. It is the most important format as ultimately is what I’m going to use for printing.

First of all you should know there is a package thesisdown with a few templates for some universities. If yours is there, or if you want to learn how are they you can have a look at the files.
On my case I didn’t have any template and the university guidelines are not long, (have two compulsory pages at the beginning and have 5 sections). That’s why I tweaked the default format inspired by the recent thesis defended on my group.

Well, without further delay let’s dive in things I learned:

Important

Captions

knitr::kable places the captions on tables at the top (by design, see issue #1189), while knitr places the captions on the bottom of figures. So if you want to have all the captions below the element you’ll need to use a different package for it (booktable, or others).

If you want to have short captions for an easy readable table of figures and table of tables you’ll need to use kable(short.caption = "TOC", caption = "Long caption below the table").

In addition, on kable if you use something like Häsler you’ll need to convert this “ä” to “\u00E4”.

I also wanted to highlight and differentiate the captions. I ended up using the caption package:

\usepackage{caption}
% Set in bold the numbering of tables and chapters
\captionsetup{labelfont=bf,width=\textwidth}

The \textwidh is to make more with the captions otherwise they just spans the size of the table or figure.

Repeating text

If you find yourself repeating some text to explain some figures, legends or tables you can use text references.

(ref:foo) Define a text reference **here**.

Then you can use (ref:foo) to repeat the same text.

Although formatting cannot be applied afterwards (i.e. **(ref:foo)**) it is handy to just write once and avoid repetition (And also if to keep backwards compatibility you can’t use the new special comment #| to specify chunk options).

Placing options

Many latex instructions go to the index.Rmd file.

The once I included are:

split_by: chapter
link-citations: true 
always_allow_html: true
colorlinks: yes
# https://bookdown.org/yihui/rmarkdown-cookbook/latex-variables.html
# links-as-notes: true # Only activate for actual printing
fontfamily: libertine
fontsize: 12pt
papersize: a4 # The printed size of the thesis
acronyms:
  loa_title: ""
  insert_loa: false
  sorting: usage
  include_unused: false
  fromfile: ./style/acronyms.yml
geometry:
 - top=25.4mm
 - bottom=25.4mm
 - left=25.4mm
 - right=25.4mm
 - bindingoffset=6.4mm
 - asymmetric
classoption: 
  - twoside
  - openright
lot: yes
lof: yes

split_by in the html format how to move to the next section.
link-citations Add a link to the citation?
colorlinks If links should have a color
links-as-notes Instead of having hyperlinks have them included as notes. It is useful for printing where the reader doesn’t have the option to click a link but might be interested in knowing more.
fontfamily and fontsize decide which font and size will be used.
papersize this chooses the available space and greatly affects the position of figures and tables, which can float on the text according to LaTeX algorithm.
acronyms Configuration of the acronyms
geometry Defines the margins, consider that on books the central zone will not be readable. The bindingoffset adds some space to make it easier reading.
classoption Options for the book format
lot and lof indicate if list of tables (lot) and list of figures (lof) should be included on the pdf output.

The book_filename if present on index.Rmd is overwritten by what is on _bookdown.yml but be careful also on what goes to _bookdown.yml and on the specific format on _output.yml .

Nice little tricks

Dedication

Looking at the source code of the bookdown book I found that the correct way was to use before_body option.

Chapter thumb

One thing I liked from other thesis is the ability to have on the printed edition a little mark on the side of the page to find a section.

My first search showed that it was possible, but I didn’t want to load the tikz package. I ended up using this solution after adding and modifying the colors, changing the size and position.

\usepackage[scale=1,angle=0,opacity=1,contents={}]{background}
\usetikzlibrary{calc}
\usepackage{ifthen}
\usepackage{lipsum}
% auxiliary counter
\newcounter{chapshift}
% the list of colors to be used (add more if needed)
\newcommand\BoxColor{%
  \ifcase\thechapshift blue!30\or red!30\or olive!30\or magenta!30\or teal!30\or lime!30\or orange!30\or violet!30\or brown!30\else yellow!30\fi}
% the main command; the mandatory argument sets the color of the vertical box
\newcommand\ChapFrame{%
  \def\TitleText{\leftmark}%
  \AddEverypageHook{%
    \ifthenelse{\isodd{\value{page}}}
      {\backgroundsetup{
        contents={%
          \begin{tikzpicture}[overlay,remember picture]
          \node[fill=\BoxColor,inner sep=0pt,rectangle,text width=1cm,
            text height=3cm,align=center,anchor=north east]
          at ($ (current page.north east) + (-0cm,- 3*\thechapshift cm) $)
          % {\rotatebox{90}{\parbox{4cm}{%
          %   \centering\textcolor{black}{\scshape\thechapshift}}}};
          {};
          \end{tikzpicture}
        }%
      }
    }
    {\backgroundsetup{
      contents={%
        \begin{tikzpicture}[overlay,remember picture]
        \node[fill=\BoxColor,inner sep=0pt,rectangle,text width=1cm,
          text height=3cm,align=center,anchor=north west]
        at ($ (current page.north west) + (-0cm,-3*\thechapshift cm) $)
        % {\rotatebox{90}{\parbox{4cm}{%
        %   \centering\textcolor{black}{\scshape\thechapshift}}}};
        {};
        \end{tikzpicture}
      }
    }
  }
\BgMaterial}%
\stepcounter{chapshift}
}

This code basically means that I need to add \ChapFrame when I want the chapter thumb (I didn’t know the name before searching this). Once started it changes colors according to \chapshift which is automatically incremented by \ChapFrame. I also set to change position according to \thechapshift so that they make a stair.

The code basically changes the position of the mark if the page is even or odd, so that it is always on the outer side of the booklet. The size is width=1cm, height=3cm with text inside it. If you want text I recommend either short titles or the chapter number \chapter to ensure it is readable.

A tiny trick I learned was to reset the counter with \afterpage{\setcounter{chapshift}{0}} after the bibliography so that the appendix would use the same mark from the beginning. If you want different colors for the appendix you could just create a new counter and a new \BoxColor

Index with index on it

I wanted to have the table of contents to show were it began, simply because with all the added page on the front and white pages it might be hard to find it. It is also handy when using the outline of the pdf version to go back the the index to then move to another section.

Simply loading the \tocbibind packages was enough:

\usepackage{tocbibind}

Screenshot of the outline and the real index. The outline has the same content as the real index.

Note: I found a “bug” were the appendix link goes to the bibliography (the previous chapter) instead of the correctly displayed page.

Table of figures and tables

It was not required but I wanted a table of tables and a table of figures, to make it easier go to results of the thesis. To add them I used the tocbibind package that automatically adds it (and the options on index.Rmd wasn’t sure from the answer I found online).

\usepackage{tocbibind}

Screenshot of the first lines of the list of figures, with a short caption for each figure

Acronyms

I repeat many acronyms on the thesis and I wanted to have a brief table with them. I used the package acronymsdown which is simple, easy and works well for web and pdf.
My only wish is that it had a way to go back to were the reader was.

To place the acronyms were I wanted I had to remove the automatic title and use the following:

# Glossary {-}

\printacronyms

I add them to the beginning after the summaries of the thesis and the preface, right before the body of the thesis.

Placing floats

I included many figures and tables which makes it hard to have all of them near where they are added on the text. While it can be forced, I didn’t want that but neither I wanted them too far away.

To avoid them going after the subsection I added this command before the title of the next subsection:

\FloatBarrier

Probably it could be done automatically renewing the subsection title format, but as I only had to do this 5 times is manageable.

Following an answer, the figure floating algorithm was set with these preferences:

\usepackage{float}
\usepackage{colortbl}
\let\origfigure\figure
\let\endorigfigure\endfigure
\renewenvironment{figure}[1][2] {
    \expandafter\origfigure\expandafter[!htbp]
} {
    \endorigfigure
}

A figure and a table on the same page, the label is in bold and the text in italics, on the right side (the page is odd) the blue chapter thumb

Github actions

To render I initially used my r-lib/actions but without using any package structure. However, once I set a DESCRIPTION file with all the package dependencies it was much faster, as I could use the setup-r-dependencies action.

Probably there is also a faster way directly installing binaries with the help of bspm or the system package manager, but this was convenient enough.

Also the action to install tinytex made my job for the pdf to render much faster (from 15 minutes to 5 minutes).

Screenshot of the github actions (on push) were bookdown-web takes ~3 minutes, bookdown-epub ~2 minutes and bookdown-pdf ~5 minutes.

Empty pages

The chapters are on the right side of the book so they must end or have a blank page before. To have a completely blank page I found a simple solution, simply load a latex package emptypage.

\usepackage{emptypage}

Title pages

Where to place titles, format it with titlesec:

\usepackage{titlesec}
\titleformat{\chapter}[display]{\fontsize{32pt}{48pt}\bfseries\sffamily\filcenter}{
    \fontsize{72pt}{72pt} \thechapter \ChapFrame
} % Content on the chapter title page
{20pt}{\lsstyle}[\thispagestyle{empty} \cleardoublepage]% https://tex.stackexchange.com/a/347162/178206
\titleformat{name=\chapter, numberless}{\normalfont\huge\bfseries\filcenter}{}{20pt}{\Huge}
\titlespacing*{\chapter}{0pt}{100pt}{40pt}

The first line load the package. Then we set the format of the chapters option display and the format of text and what appears on that page. \thechapter is the title of the chapter (while \chatper is the number). \chapFrame is the new command defined to set chapter thumb.

The other benefit this had was that the title page had no page number.

A screenshot of the introduction title page. There is a number 1 and Introduction a couple of lines below. On the right side a red box 3 cm below the top.

Page numbers

I decided to use different style for page numbers

\usepackage[automark,headsepline]{scrlayer-scrpage}% sets page style scrheadings automatically
\clearpairofpagestyles
\ihead{\leftmark}
\ohead*{\pagemark}
\setkomafont{pagenumber}{}% default is \normalfont\normalcolor

Include pdfs

As part of the appendix I added my publications on their pdf format. To do so I used the following code modified from the original answer:

\usepackage{pdfpages} 
\includepdf[pages=-, pagecommand={}, templatesize={\textwidth}{\textheight  - 25pt}, trim=0 0 0 20pt,]{pdf/paper.pdf}

I had to tweak the size were it was placed to keep the page numbers and running titles. The last curly brackets indicate the location of the pdf to include.

The benefit of this is that these pages now are also numbered with the thesis style and the title.

Screenshot of the pdf included showing the page number of the thesis and the content of the article “Multi-omic modelling of inflammatory bowel disease with regularized canonical correlation analysis”

Running titles

If you have a long title such as you are probably interested on having a shorter version for it on the thesis pages.

## experDesign: stratifying samples into batches with minimal bias

\sectionmark{experDesign: paper}

I don’t think it actually made a big difference but it might we important for chapters.

Merging pdfs

As I said I needed some pages at the beginning of the thesis. But I wanted to keep the outline of the pdf, and I found an solution online explaining how to do it.

To add multiple pdf I did this:

gs -q -SDEVICE=pdfwrite -DPDFSETTINGS=/prepress -o merged.pdf page1.pdf page2.pdf thesis.pdf

Reducing pdf size

At the end the pdf was bigger than what I could send over email.

I found this answer that helped me to reduce the size and send it.

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Conclusion

The process of writing the thesis is usually one of the last steps on a PhD. I recommend to write something and avoid having a blank page. But once written you must take care of the presentation and style, this is a complete different skill than writing or research, so it can be specially exhausting.

Bookdown through the preamble and body options is great for setting your style. But if you are short of time, are tired you might benefit from working and using some of these already created solutions and just modify what you need (as I did).

To finish, so that you can see the final format it is here. There you can download it in pdf too to see most of these commands in action.

If you are writing your thesis, enjoy, keep calm and reuse other solutions!