rtweet | B101nfo

useR madrid: rtweet

Thu, 29 Feb 2024 19:00:00 +0200

This presentation was in Spanish. I shared the history of my involvement with rtweet and what is happening with the package and Twitter API.

New rtweet release: 2.0.0

Fri, 16 Feb 2024 00:00:00 +0000

This is a brief announcement of rtweet version 2.0.0. This major version changes signals the move from the API v1.1 to the API v2.

There haven’t been many changes since 1.2.1 but this is to signal that the API v1.1 is deprecated.

The previous release was a bit of a rush to meet with the requirements of CRAN maintainers to fix an error and it wasn’t polished. Some users complained that it was difficult to find what worked. In this release I focused mostly to make life easier for users:

Now there is a document the deprecated functions from API v1.1 to API v2: see help("rtweet-deprecated", "rtweet"). I also made it easier for the rtweet to work with API v2: the release of httr2 1.0.0 version helped to avoid some workarounds with the authentication process.

I also focused on updating the vignettes to the most up to date recommendations. I am not sure the streaming vignettes is up to date (but keep reading why I left it as is).

Last, following CRAN policy: if users create rtweet data they can now delete it with client_clean() and auth_clean().

Future releases

For the last year I asked the community for a co-maintainer with interest in the package. Unfortunately, people that showed some interest at the end didn’t commit to it.

At the same time I also asked for donations to support an API access. It currently costs 100€ to access most endpoints which is needed to test and develop the package. However, this is more than half of what I spend in groceries last month.
Other packages like academictwitteR are also stopping development/support. Although not archived from CRAN, it has a note in the README:

Note this repo is now ARCHVIED due to changes to the Twitter API. The paid API means open-source development of this package is no longer feasible.

Similarly without financial help and community interest I won’t invest more time on it.
This is the last version that I release. I have other interests and I would like to focus on other projects. My focus will be on updating and releasing some packages I have. I also want to focus more on my own company to help the R community (and beyond). I will write about the company shortly.

There have been some discussions on social media how to signal deprecation of packages. The only method available on CRAN that I know is to declare a package ORPHANATED. I have requested to CRAN to declared the package ORPHANATED.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.3.1 (2023-06-16)
##  os       Ubuntu 22.04.4 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language en
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2024-02-24
##  pandoc   3.1.1 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date (UTC) lib source
##  blogdown      1.18    2023-06-19 [1] CRAN (R 4.3.1)
##  bookdown      0.37    2023-12-01 [1] CRAN (R 4.3.1)
##  bslib         0.6.1   2023-11-28 [1] CRAN (R 4.3.1)
##  cachem        1.0.8   2023-05-01 [1] CRAN (R 4.3.1)
##  cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.1)
##  digest        0.6.34  2024-01-11 [1] CRAN (R 4.3.1)
##  evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.2)
##  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.1)
##  htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.2)
##  jquerylib     0.1.4   2021-04-26 [1] CRAN (R 4.3.1)
##  jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.3.1)
##  knitr         1.45    2023-10-30 [1] CRAN (R 4.3.2)
##  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.2)
##  R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.1)
##  rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.1)
##  rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.1)
##  rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
##  sass          0.4.8   2023-12-06 [1] CRAN (R 4.3.1)
##  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.1)
##  xfun          0.42    2024-02-08 [1] CRAN (R 4.3.1)
##  yaml          2.3.8   2023-12-11 [1] CRAN (R 4.3.1)
## 
##  [1] /home/lluis/bin/R/4.3.1
##  [2] /opt/R/4.3.1/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Releasing rtweet 1.2.0

Mon, 20 Mar 2023 00:00:00 +0000

I’m very excited to announce that rtweet 1.2.0 is now available on GitHub! Install it by running:

devtools::install_github("ropensci/rtweet")

Then load it in a fresh session with:

library(rtweet)

New features

This version adds many new endpoints to retrieve data from twitter:

From lists
From tweets
About users
Also, about statistics of your own content.

You can read about them in the NEWS.

Authentication

Besides fixing a problem preventing new users to use auth_setup_default(), in this release there is a new authentication mechanism.

Some endpoints require a new authentication method not previously used by rtweet. This authentication mechanism requires setting up a client.
To support it, I have added some functions to create it, save it, and use it modelling the functions from auth_*. There is now one client provided by rtweet if you don’t want to configure your own:

client_setup_default()

Additionally, I briefly expanded the authentication vignette (vignette("auth", "rtweet")) to include a section about how to obtain the required credentials. Once you get them is pretty straight forward:

auth_oauth2 <- rtweet_oauth2(app = "my_awesome_app")

This mechanism is required by some functions which are of special interest: user_self(), tweet_bookmarked(), user_blocked(), and user_timeline().

Note that due to upstream reasons, the authentication is only valid for 2 hours. You will be asked to approve the client again after the 2 hours (and save it again!).

We can set the authentication as we usually do:

auth_as(auth_oauth2)

And start retrieving our data in Twitter!

New functions

me <- user_self()
bookmarked <- user_bookmarks(me$id, n = 120)

rtweet will make as many requests as needed and automatically paginate the results. However, if you try this you might realize that the queries are slow. These are the limits imposed by Twitter.

If you want to keep track of the progress of your query, you can use verbose = TRUE:

blocked <- user_blocked(me$id, n = Inf, verbose = TRUE)
timeline <- user_timeline(me$id, n = 800, verbose = TRUE)

It will also store the data of the requests in a temporary file, in case you lose the connection you can still recover it.

Some endpoints have a length limit on the accepted input:

bioconductor <- user_by_username("Bioconductor")
bioconductor_followers <- user_followers(bioconductor$id, n = 200)
us <- user_search(ids = bioconductor_followers$id)

Errors are, in principle, easier to understand in these new functions, thanks to the messages provided:

user_blocked(bioconductor$id, n = Inf, verbose = TRUE)

Other

These new endpoints provide access to many data which only the default information is converted to a nice table. If you request more data, via expansions and fields: replies, information about the user of a tweet, … you will have to wait next release.
You can already get them but with parse = FALSE. My intention was to provide more parsing support in this release, but I think it is better to make more releases more often.

The API also provides an endpoint to check if data stored is compliant with the Terms of Service. I started working on these endpoints after the streaming endpoints because they are important.

Side story

With the deprecation of the streaming endpoints and the function stream_tweets I implemented the first three functions using Twitter’s API v2. They use a bearer token as authentication mechanism.

Many endpoints of API v2 also use this authentication mechanism and made it easy to support them.

But there was a petition to retrieve the bookmarked tweets. That endpoint required OAuth2 mechanism and didn’t allow the use of the bearer token. It is relevant because bookmarks are not provided with the data dump you can request from Twitter. This endpoint is the only automatic way to retrieve them from Twitter if you used them!

Final

This update is the last update of rtweet. The new API plans make it impossible to continue developing and testing software like rtweet without substantial financial investment (at least USD$100/month).

More importantly, this will restrict who can use the package. I think those few users still using rtweet might also afford to pay for support or development of new features. If you are one of them you can sponsor my work in rtweet.

I will remove the package from CRAN one month after the new API enters into effect (~ 1st July). Farewell Twitter.

rtweet future

Thu, 16 Feb 2023 00:00:00 +0000

Background: how I became the maintainer of rtweet

I didn’t want to maintain rtweet. It might sound strange coming from its maintainer, but I didn’t want the responsibility of writting software 12k people install it monthly. My offer was always to help with it so that the users could benefit from improvements and bug fixes on the package. I initially thought that having permissions to close issues, label them would help the community and the maintainer.

I was not totally altruistic. As recently recommended by Maëlle and Steffi, I had some interest in the package. I had a bot posting some plots daily. I wanted to have alt text in the tweets. There was a pending PR for just that. I could fork and use my own version or help with the package.

I got in touch with rOpenSci about the package. After some time waiting hearing back from the author of the package (Many thanks Michael W. Kearney, I was given edit permissions to the repository (under the helping eye of Scott Chamberlain). At the time, I got permission from rOpenSci there were 167 issues and PR open.

As I started going through them and changing the source code, people showed up to help (Thanks!). New contributors helped with closing issues via new PR or simply advising about naming functions or asking for its future. Another developer gained access to repository and contributed with their expertise (resulting in some funny moments). Despite all (breaking) changes, we tried to make the package easier to maintain. After waiting some time, for further community feedback, I released a new version through CRAN.

Shortly after, for some reasons my bot got less views and engagement. It made me realize that it might be an opportunity to start a professional project around the content of what this bot did. In addition, although I learn a lot from the online communities I decided to connect more with the local community. I am currently involved with re-launching the Barcelona R user group and organizing the Spanish R conference. These activities take time an energy away from package maintenance (not just rtweet). However, my support for the users of rtweet remains.

The community support I received was great. I hope to pass on it. CC-BY-4.0 Artwork by @allison_horst.

Current situation: supporting the v1 and adding support to the v2 API

To maintain rtweet I needed to adapt and understand the Twitter’s API (v1) documentation. The documentation didn’t always match the functionality. rtweet follows the API tightly, any change might break the package. For instance, the addition of the edit field broke the parsing of the tweets.

At the same time one of the oldest issues in rtweet is moving to the new API v2. With the 1.0 release, I decided it was time to stop adding features relying in it. It was clear that Twitter was moving to deprecate and replace it with the newer API. There are many benefits of the new API to developer and users. But as I realized that I should still fix bugs related to them.

The 1.1 release have recently provided some bug fixes and support for the v2. There is also an outstanding issue preventting new users of connecting to Twitter (fixed in github). The streaming endpoint in v1 stopped working before the announced date. So the new release included support for the replacement endpoint in v2.

As you have guessed, between releases I had been thinking and working to support API v2. The foundations to support the new endpoints were already set and easily expanded to new endpoints. In the development version of the package (in the devel branch), it is possible to retrieve bookmarks, retrieve the archive if you have academic access, among other endpoints.

Going forward: supporting rtweet users of v2 API

Recently (~3 weeks), there have been anouncements of future changes in who and how will be able to access the API. Simultaneously, there have been unannounced restrictions affecting other tools using the API. Recent changes and announcements are driven by need of money to sustain Twitter. To maintain rtweet with its current functionality it might need funds.

I will continue supporting rtweet and the freely accessible endpoints. But given that I will have less time and energy for rtweet, I am looking for a co-maintainer to help me:

Supporting new endpoints, using httr2, testing an API in CI, …
Review changes to avoid new bugs.
Help with issues and questions that the transition to API v2 and the current uncertainty bring.

Get in touch with me in this issue.

From time to time, I receive bug reports and petitions from premium API users. Currently, premium users get access to more data, elevated request rates and other endpoints. Helping them is usually challenging, and developing support for these endpoints will be difficult. To continue supporting these users I’ve set up a ☕ buymeacoffee and I, and my co-maintainer, will be open to consulting jobs about rtweet. With these jobs and funding we will be able to support them and implement new endpoints for all the users.

There is scarce information about the API changes and prices. Changes might come suddenly. We’ll do our best to keep up and inform all users. I hope this will be a good way to continue supporting the package an the community of users. Let me know if you have other suggestions.

Decorating/contributing to R. CC-BY-4.0 artwork by @allison_horst.

Thanks Maëlle for your support and reviewing the post.

Upgrading rtweet to 1.0.2

Mon, 04 Jul 2022 00:00:00 +0000

In this post I will provide some examples of what has changed between rtweet 0.7.0 and rtweet 1.0.2. I hope both the changes and this guide will help all users. I highlight the most important and interesting changes in this blog post, and for a full list of changes you can consult it on the NEWS.

Big breaking changes

More consistent output

This is probably what will affect the most users. All functions that return data about tweets¹ now return the same columns.

For example if we search some tweets we’ll get the following columns:

> tweets <- search_tweets("weather")
> colnames(tweets)
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "full_text"                    
 [5] "truncated"                     "display_text_range"           
 [7] "entities"                      "metadata"                     
 [9] "source"                        "in_reply_to_status_id"        
[11] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[13] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[15] "geo"                           "coordinates"                  
[17] "place"                         "contributors"                 
[19] "is_quote_status"               "retweet_count"                
[21] "favorite_count"                "favorited"                    
[23] "retweeted"                     "lang"                         
[25] "quoted_status_id"              "quoted_status_id_str"         
[27] "quoted_status"                 "possibly_sensitive"           
[29] "retweeted_status"              "text"                         
[31] "favorited_by"                  "scopes"                       
[33] "display_text_width"            "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "query"                         "withheld_scope"               
[41] "withheld_copyright"            "withheld_in_countries"        
[43] "possibly_sensitive_appealable"

rtweet now minimizes the processing of tweets and only returns the same data as provided by the API while making it easier to handle by R. However, to preserve the nested nature of the data returned some fields are now nested inside other. For example, previously fields "bbpx_coords", "geo_coords", "coords_coords" were returned as separate columns, but they are now nested inside "place", "coordinates" or "geo" depending where they are provided. Some columns previously calculated by rtweet are now not returned, like "rtweet_favorite_count". At the same time it provides with new columns about each tweet like the "withheld_*" columns.

If you scanned through the columns you might have noticed that columns "user_id" and "screen_name" are no longer returned. This data is still returned by the API but it is now made available to the rtweet users via users_data():

> colnames(users_data(tweets))
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

This blog post should help you find the right data columns, but if you don’t find what you are looking for it might be nested inside a column.
Try using dplyr::glimpse() to explore the data and locate nested columns. For example the entities column (which is present in both tweets and users) have the following useful columns:

> names(tweets$entities[[1]])
[1] "hashtags"      "symbols"       "user_mentions" "urls"         
[5] "media"

Similarly if you look up a user via search_users() or lookup_users() you’ll get consistent data:

> users <- lookup_users(c("twitter", "rladiesglobal", "_R_Foundation"))
> colnames(users)
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

You can use tweets_data() to retrieve information about their latest tweet:

> colnames(tweets_data(users))
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "text"                         
 [5] "truncated"                     "entities"                     
 [7] "source"                        "in_reply_to_status_id"        
 [9] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[11] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[13] "geo"                           "coordinates"                  
[15] "place"                         "contributors"                 
[17] "is_quote_status"               "retweet_count"                
[19] "favorite_count"                "favorited"                    
[21] "retweeted"                     "lang"                         
[23] "retweeted_status"              "possibly_sensitive"           
[25] "quoted_status"                 "display_text_width"           
[27] "user"                          "full_text"                    
[29] "favorited_by"                  "scopes"                       
[31] "display_text_range"            "quoted_status_id"             
[33] "quoted_status_id_str"          "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "metadata"                      "query"                        
[41] "withheld_scope"                "withheld_copyright"           
[43] "withheld_in_countries"         "possibly_sensitive_appealable"

You can merge them via:

users_and_last_tweets <- cbind(users, id_str = tweets_data(users)[, "id_str"])

In the future (see below), with helper functions managing the output of rtweet will become easier.

Finally, get_followers() and get_friends() now return the same columns:

> colnames(get_followers("_R_Foundation"))
[1] "from_id" "to_id"  
> colnames(get_friends("_R_Foundation"))
[1] "from_id" "to_id"

This will make it easier to build networks of connections (although you might want to convert screen names to ids or vice versa).

More consistent interface

All paginated functions that don’t return tweets now use a consistent pagination interface (except the premium endpoints). They all store the “next cursor” in an rtweet_cursor attribute, which will be automatically retrieved when you use the cursor argument. This will make it easier to continue a query you started:

users <- get_followers("_R_Foundation")
users
     
# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

They support max_id and since_id to find earlier and later tweets respectively:

# Retrieve all the tweets made since the previous request
newer <- search_tweets("weather", since_id = tweets)
# Retrieve tweets made before the previous request
older <- search_tweets("weather", max_id = tweets)

If you want more tweets than it is allowed by the rate limits of the API, you can use retryonratelimit to wait as long as needed:

long <- search_tweets("weather", n = 1000, retryonratelimit = TRUE)

This will keep busy your terminal until the 1000 tweets are retrieved.

Saving data

An unexpected consequence of returning more data (now matching that returned by the API) is that it is harder to save it in a tabular format. For instance one tweet might have one media, mention two users and have three hashtags. There isn’t a simple way to save it in a single row uniformly for all tweets or it could lead to confusion.

This resulted in deprecating save_as_csv, read_twitter_csv and related functions because they don’t work with the new data structure and it won’t be possible to load the complete data from a csv. They will be removed in later versions.

Many users will benefit from saving to RDS (e.g., saveRDS() or readr::write_rds()), and those wanting to export to tabular format can simplify the data to include only that of interest before saving with generic R functions (e.g., write.csv() or readr::write_csv()).

Other breaking changes

Accessibility is important and for this reason if you tweet via post_tweet() and add an image, gif or video you’ll need to provide the media alternative text. Without media_alt_text it will not allow you to post.
tweet_shot() has been deprecated as it no longer works correctly. It might be possible to bring it back, but the code is complex and I do not understand enough to maintain it. If you’re interested in seeing this feature return, checkout the discussion about this issue and let me know if you have any suggestions.
rtweet also used to provide functions for data on emojis, langs and stopwordslangs. These are useful resources for text mining in general - not only in tweets - however they need to be updated to be helpful and would be better placed in other packages, for instance emojis is now on the bdpar package. Therefore they are no longer available in rtweet.
The functions like suggested_*() have been removed as they have been broken since 2019.

Easier authentication

An exciting part of this release has been a big rewrite of the authentication protocol. While it is compatible with previous rtweet authentication methods it has also some important new functions which make it easier to work with rtweet and the twitter API in different ways.

Different ways to authenticate

If you just want to test the package, use the default authentication auth_setup_default() that comes with rtweet. If you use it for one or two days you won’t notice any problem.

If you want to use the package for more than a couple of days, I recommend you set up your own token via rtweet_user(). It will open a window to authenticate via the authenticated account in your default browser. This authentication won’t allow you to do everything but it will avoid running out of requests and being rate-limited.

If you plan to make heavy use of the package, I recommend registering yourself as developer and using one of the following two mechanisms, depending on your plans:

Collect data and analyze: rtweet_app().
Set up a bot: rtweet_bot()

Find more information in the Authentication with rtweet vignette.

Storing credentials

Previously rtweet saved each token created, but now non-default tokens are only saved if you ask. You can save them manually via auth_save(token, "my_app"). Bonus, if you name your token as default (auth_save(token, "default")) it will be used automatically upon loading the library.

Further, tokens are now saved in the location output by tools::R_user_dir("rtweet", "config"), rather than in your home directory. If you have previous tokens saved or problems identifying which token is which use auth_sitrep(). This will provides clues to which tokens might be duplicated or misconfigured but it won’t check if they work. It will also automatically move your tokens to the new path.

To check which credentials you have stored use auth_list() and load them via auth_as("my_app"). All the rtweet functions will use the latest token loaded with auth_as (unless you manually specify one when calling it). If you are not sure which token you are using you can use auth_get() it will return the token in use, list them or ask you to authenticate.

Other changes of note

This is a list of other changes that aren’t too big or are not breaking changes but are worthy enough of a mention:

Iteration and continuation of requests

Using cursors, pagination or waiting until you can make more queries is now easier. For example you can continue previous requests via:

users <- get_followers("_R_Foundation")
users

# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

Additions

There is now a function to find a thread of a user. You can start from any tweet and it will find all the tweets of the thread: tweet_threading("1461776330584956929").

There is a lot of interest in downloading and keeping track of interactions on Twitter. The amount of interest is big enough that Twitter is releasing a new API to provide more information of this nature.

Future

Twitter API v2 is being released and soon it will replace API v1. rtweet up to now, including this release, uses API v1 so it will need to adapt to the new endpoints and new data returned.

First will be the streaming endpoints in November, so expect more (breaking?) changes around those dates if not earlier.

I would also like to make it easier for users, dependencies and the package itself to handle the outputs. To this regard I would like to provide some classes to handle the different type of objects it returns.

This will help avoid some of the current shortcomings. Specifically I would like to provide functions to make it easier to reply to previous tweets, extract nested data, and subset tweets and the accompanying user information.

Conclusions

While I made many breaking changes I hope these changes will smooth future development and help both users and maintainers.

Feel free to ask on the rOpenSci community if you have questions about the transition or find something amiss. Please let me know! It will help me prioritize which endpoints are more relevant to the community. (And yes, the academic archive endpoint is on the radar.)

It is also possible that I overlooked something and I thought the code is working when it isn’t. For example, after several months of changing the way the API is parsed, several users found it wasn’t handling some elements. Let me know of such or similar cases and I’ll try to fix it.

In case you find a bug, check the open issues and if it has not already been reported, open an issue on GitHub. Don’t forget to make a reprex and if possible provide the id of the tweets you are having trouble with. Unfortunately it has happened that when I came to look at a bug I couldn’t reproduce it as I wasn’t able to find the tweet which caused the error.

This release includes contributions from Hadely Wicham, Bob Rudis, Alex Hayes, Simon Heß, Diego Hernán, Michael Chirico, Jonathan Sidi, Jon Harmon, Andrew Fraser and many other that reported bugs or provided feedback. Many thanks all for using it, your interest to keep it working and improving rtweet for all.

Finally, you can read the whole NEWS online and the examples.

Happy tweeting!

Acknowledgements

This is a repost of the entry for rOpenSci. The post was edited and improved by Yanina Bellini Saibene and Steffi LaZerte, the community manager and assistant. Many thanks

Specifically these: get_favorites(), get_favorites_user(), get_mentions(), get_my_timeline(), get_retweets(), get_timeline(), get_timeline_user(), lists_statuses(), lookup_statuses(), lookup_tweets(), search_30day(), search_fullarchive(), search_tweets(), tweet_shot() and tweet_threading().↩︎