API | B101nfo

Upgrading rtweet to 1.0.2

Mon, 04 Jul 2022 00:00:00 +0000

In this post I will provide some examples of what has changed between rtweet 0.7.0 and rtweet 1.0.2. I hope both the changes and this guide will help all users. I highlight the most important and interesting changes in this blog post, and for a full list of changes you can consult it on the NEWS.

Big breaking changes

More consistent output

This is probably what will affect the most users. All functions that return data about tweets¹ now return the same columns.

For example if we search some tweets we’ll get the following columns:

> tweets <- search_tweets("weather")
> colnames(tweets)
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "full_text"                    
 [5] "truncated"                     "display_text_range"           
 [7] "entities"                      "metadata"                     
 [9] "source"                        "in_reply_to_status_id"        
[11] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[13] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[15] "geo"                           "coordinates"                  
[17] "place"                         "contributors"                 
[19] "is_quote_status"               "retweet_count"                
[21] "favorite_count"                "favorited"                    
[23] "retweeted"                     "lang"                         
[25] "quoted_status_id"              "quoted_status_id_str"         
[27] "quoted_status"                 "possibly_sensitive"           
[29] "retweeted_status"              "text"                         
[31] "favorited_by"                  "scopes"                       
[33] "display_text_width"            "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "query"                         "withheld_scope"               
[41] "withheld_copyright"            "withheld_in_countries"        
[43] "possibly_sensitive_appealable"

rtweet now minimizes the processing of tweets and only returns the same data as provided by the API while making it easier to handle by R. However, to preserve the nested nature of the data returned some fields are now nested inside other. For example, previously fields "bbpx_coords", "geo_coords", "coords_coords" were returned as separate columns, but they are now nested inside "place", "coordinates" or "geo" depending where they are provided. Some columns previously calculated by rtweet are now not returned, like "rtweet_favorite_count". At the same time it provides with new columns about each tweet like the "withheld_*" columns.

If you scanned through the columns you might have noticed that columns "user_id" and "screen_name" are no longer returned. This data is still returned by the API but it is now made available to the rtweet users via users_data():

> colnames(users_data(tweets))
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

This blog post should help you find the right data columns, but if you don’t find what you are looking for it might be nested inside a column.
Try using dplyr::glimpse() to explore the data and locate nested columns. For example the entities column (which is present in both tweets and users) have the following useful columns:

> names(tweets$entities[[1]])
[1] "hashtags"      "symbols"       "user_mentions" "urls"         
[5] "media"

Similarly if you look up a user via search_users() or lookup_users() you’ll get consistent data:

> users <- lookup_users(c("twitter", "rladiesglobal", "_R_Foundation"))
> colnames(users)
 [1] "id"                      "id_str"                 
 [3] "name"                    "screen_name"            
 [5] "location"                "description"            
 [7] "url"                     "protected"              
 [9] "followers_count"         "friends_count"          
[11] "listed_count"            "created_at"             
[13] "favourites_count"        "verified"               
[15] "statuses_count"          "profile_image_url_https"
[17] "profile_banner_url"      "default_profile"        
[19] "default_profile_image"   "withheld_in_countries"  
[21] "derived"                 "withheld_scope"         
[23] "entities"

You can use tweets_data() to retrieve information about their latest tweet:

> colnames(tweets_data(users))
 [1] "created_at"                    "id"                           
 [3] "id_str"                        "text"                         
 [5] "truncated"                     "entities"                     
 [7] "source"                        "in_reply_to_status_id"        
 [9] "in_reply_to_status_id_str"     "in_reply_to_user_id"          
[11] "in_reply_to_user_id_str"       "in_reply_to_screen_name"      
[13] "geo"                           "coordinates"                  
[15] "place"                         "contributors"                 
[17] "is_quote_status"               "retweet_count"                
[19] "favorite_count"                "favorited"                    
[21] "retweeted"                     "lang"                         
[23] "retweeted_status"              "possibly_sensitive"           
[25] "quoted_status"                 "display_text_width"           
[27] "user"                          "full_text"                    
[29] "favorited_by"                  "scopes"                       
[31] "display_text_range"            "quoted_status_id"             
[33] "quoted_status_id_str"          "quoted_status_permalink"      
[35] "quote_count"                   "timestamp_ms"                 
[37] "reply_count"                   "filter_level"                 
[39] "metadata"                      "query"                        
[41] "withheld_scope"                "withheld_copyright"           
[43] "withheld_in_countries"         "possibly_sensitive_appealable"

You can merge them via:

users_and_last_tweets <- cbind(users, id_str = tweets_data(users)[, "id_str"])

In the future (see below), with helper functions managing the output of rtweet will become easier.

Finally, get_followers() and get_friends() now return the same columns:

> colnames(get_followers("_R_Foundation"))
[1] "from_id" "to_id"  
> colnames(get_friends("_R_Foundation"))
[1] "from_id" "to_id"

This will make it easier to build networks of connections (although you might want to convert screen names to ids or vice versa).

More consistent interface

All paginated functions that don’t return tweets now use a consistent pagination interface (except the premium endpoints). They all store the “next cursor” in an rtweet_cursor attribute, which will be automatically retrieved when you use the cursor argument. This will make it easier to continue a query you started:

users <- get_followers("_R_Foundation")
users
     
# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

They support max_id and since_id to find earlier and later tweets respectively:

# Retrieve all the tweets made since the previous request
newer <- search_tweets("weather", since_id = tweets)
# Retrieve tweets made before the previous request
older <- search_tweets("weather", max_id = tweets)

If you want more tweets than it is allowed by the rate limits of the API, you can use retryonratelimit to wait as long as needed:

long <- search_tweets("weather", n = 1000, retryonratelimit = TRUE)

This will keep busy your terminal until the 1000 tweets are retrieved.

Saving data

An unexpected consequence of returning more data (now matching that returned by the API) is that it is harder to save it in a tabular format. For instance one tweet might have one media, mention two users and have three hashtags. There isn’t a simple way to save it in a single row uniformly for all tweets or it could lead to confusion.

This resulted in deprecating save_as_csv, read_twitter_csv and related functions because they don’t work with the new data structure and it won’t be possible to load the complete data from a csv. They will be removed in later versions.

Many users will benefit from saving to RDS (e.g., saveRDS() or readr::write_rds()), and those wanting to export to tabular format can simplify the data to include only that of interest before saving with generic R functions (e.g., write.csv() or readr::write_csv()).

Other breaking changes

Accessibility is important and for this reason if you tweet via post_tweet() and add an image, gif or video you’ll need to provide the media alternative text. Without media_alt_text it will not allow you to post.
tweet_shot() has been deprecated as it no longer works correctly. It might be possible to bring it back, but the code is complex and I do not understand enough to maintain it. If you’re interested in seeing this feature return, checkout the discussion about this issue and let me know if you have any suggestions.
rtweet also used to provide functions for data on emojis, langs and stopwordslangs. These are useful resources for text mining in general - not only in tweets - however they need to be updated to be helpful and would be better placed in other packages, for instance emojis is now on the bdpar package. Therefore they are no longer available in rtweet.
The functions like suggested_*() have been removed as they have been broken since 2019.

Easier authentication

An exciting part of this release has been a big rewrite of the authentication protocol. While it is compatible with previous rtweet authentication methods it has also some important new functions which make it easier to work with rtweet and the twitter API in different ways.

Different ways to authenticate

If you just want to test the package, use the default authentication auth_setup_default() that comes with rtweet. If you use it for one or two days you won’t notice any problem.

If you want to use the package for more than a couple of days, I recommend you set up your own token via rtweet_user(). It will open a window to authenticate via the authenticated account in your default browser. This authentication won’t allow you to do everything but it will avoid running out of requests and being rate-limited.

If you plan to make heavy use of the package, I recommend registering yourself as developer and using one of the following two mechanisms, depending on your plans:

Collect data and analyze: rtweet_app().
Set up a bot: rtweet_bot()

Find more information in the Authentication with rtweet vignette.

Storing credentials

Previously rtweet saved each token created, but now non-default tokens are only saved if you ask. You can save them manually via auth_save(token, "my_app"). Bonus, if you name your token as default (auth_save(token, "default")) it will be used automatically upon loading the library.

Further, tokens are now saved in the location output by tools::R_user_dir("rtweet", "config"), rather than in your home directory. If you have previous tokens saved or problems identifying which token is which use auth_sitrep(). This will provides clues to which tokens might be duplicated or misconfigured but it won’t check if they work. It will also automatically move your tokens to the new path.

To check which credentials you have stored use auth_list() and load them via auth_as("my_app"). All the rtweet functions will use the latest token loaded with auth_as (unless you manually specify one when calling it). If you are not sure which token you are using you can use auth_get() it will return the token in use, list them or ask you to authenticate.

Other changes of note

This is a list of other changes that aren’t too big or are not breaking changes but are worthy enough of a mention:

Iteration and continuation of requests

Using cursors, pagination or waiting until you can make more queries is now easier. For example you can continue previous requests via:

users <- get_followers("_R_Foundation")
users

# use `cursor` to find the next "page" of results
more_users <- get_followers("_R_Foundation", cursor = users)

Additions

There is now a function to find a thread of a user. You can start from any tweet and it will find all the tweets of the thread: tweet_threading("1461776330584956929").

There is a lot of interest in downloading and keeping track of interactions on Twitter. The amount of interest is big enough that Twitter is releasing a new API to provide more information of this nature.

Future

Twitter API v2 is being released and soon it will replace API v1. rtweet up to now, including this release, uses API v1 so it will need to adapt to the new endpoints and new data returned.

First will be the streaming endpoints in November, so expect more (breaking?) changes around those dates if not earlier.

I would also like to make it easier for users, dependencies and the package itself to handle the outputs. To this regard I would like to provide some classes to handle the different type of objects it returns.

This will help avoid some of the current shortcomings. Specifically I would like to provide functions to make it easier to reply to previous tweets, extract nested data, and subset tweets and the accompanying user information.

Conclusions

While I made many breaking changes I hope these changes will smooth future development and help both users and maintainers.

Feel free to ask on the rOpenSci community if you have questions about the transition or find something amiss. Please let me know! It will help me prioritize which endpoints are more relevant to the community. (And yes, the academic archive endpoint is on the radar.)

It is also possible that I overlooked something and I thought the code is working when it isn’t. For example, after several months of changing the way the API is parsed, several users found it wasn’t handling some elements. Let me know of such or similar cases and I’ll try to fix it.

In case you find a bug, check the open issues and if it has not already been reported, open an issue on GitHub. Don’t forget to make a reprex and if possible provide the id of the tweets you are having trouble with. Unfortunately it has happened that when I came to look at a bug I couldn’t reproduce it as I wasn’t able to find the tweet which caused the error.

This release includes contributions from Hadely Wicham, Bob Rudis, Alex Hayes, Simon Heß, Diego Hernán, Michael Chirico, Jonathan Sidi, Jon Harmon, Andrew Fraser and many other that reported bugs or provided feedback. Many thanks all for using it, your interest to keep it working and improving rtweet for all.

Finally, you can read the whole NEWS online and the examples.

Happy tweeting!

Acknowledgements

This is a repost of the entry for rOpenSci. The post was edited and improved by Yanina Bellini Saibene and Steffi LaZerte, the community manager and assistant. Many thanks

Specifically these: get_favorites(), get_favorites_user(), get_mentions(), get_my_timeline(), get_retweets(), get_timeline(), get_timeline_user(), lists_statuses(), lookup_statuses(), lookup_tweets(), search_30day(), search_fullarchive(), search_tweets(), tweet_shot() and tweet_threading().↩︎

Bugs in R

Tue, 16 Nov 2021 00:00:00 +0000

This post has a relatively long introduction, you can skip and go ahead to the analysis.

I knew about the r-devel (abbreviated Rd) mailing list that is where some discussions about the language happen and I had read that post of the R core requesting help reviewing bug reports and I had requested an account the same day it came out to be able to post on Bugzilla: https://bugs.r-project.org/. But I hadn’t reported any bug or anything, what had I to bring?

After RStudio 2021 I saw announced the https://r-devel.slack.com (you can join via this website) and when I joined I checked some bugs and I found something odd. This lead to the creation of my first R bug: 18055.

Then I thinking about the analysis about package reviews I realized it was analyzing issues. Then it came to me that I could actually also analyse the issues of R, aka bugs reports.

Collecting the data

First was step was collecting the data needed. Similar with the rOpenSci and Bioconductor analysis I knew I might need to create a package or a script just to retrieve the data.

I found somehow that it reports some data as xml and I though I could use that. But exploring the documentation I found it had an API that could be used to retrieve data. It required authentication to interact with the API. Instead of putting me off it made it a reasonable challenge and progression. If previously I used the gh package to authenticate and retrieve the raw data this time I had to learn how to authenticate to an API. I already developed a package that uses a poorly explained API to retrieve documents and adding a new step to authenticate requests was a small enough step.

I also found a package, bugtractr, that did this already but I had some problems using it and didn’t use authenticated requests to get the data. This meant it couldn’t retrieve all data I wanted. So I went on developing my own package to interact with Bugzilla’s API.

As I was learning to interact with API and I wanted to make it useful for the R community I looked at how to make that. Lukyly for me at the time I was doing this, I found the book HTTP testing in R, which was still under heavy writing but almost complete, and started reading and using its helpful advice to learn. One of the first recommendations was to contact with the API providers, so I email the R core about my intentions.

They raised some concerns: - Impact on the load of the machine. - if the API is robust enough. - However, semi-automating report submissions needs more thought.

Yes, at that time I had realized that the API allows to submit bug reports on the database (comments and attachments too), so I thought it could be an easy way to help people submit more bugs: submitting bugs from R itself.

After some exchanges about why, and how I was trying to retrieve data from Bugzilla I was deferred Simon Urbanek.

By that time I already posted about this on R-devel and got some interest from R contributing working group to which I presented the idea on March 12 (One month after the first commit on the package).

It was suggested on that meeting to present a Google Summer of Code¹ project which closed the project submission period shortly after. Soon two students contacted me and Heather Turner, who agreed to co-mentor the project, to write a proposal to work on my project.

By that time Simon kindly provided a database dump (without the user list) due to concerns on privacy and load on the server (which I found it can return different results on the same query), and provided the id of R core members.

The analysis

This is a mix of analysis for three several purposes:

To understand what is going on with bug reports.
To understand how to make better bug reports to help bug submissions via bugRzilla.
To help the R contributor working group and R forwards to identify contributors.
To help the R core team identifying possible areas of improvements.

It is based on the analysis by Piyush Kumar and myself. If you want code and explore the database follow these links and you’ll be able to download it.

A first exploration is to see the bug ids and the creation time of the bug id:

The first suprising thing are these three points that appear outside the line the other bugs form.

One of these outliers is a test when the bug reports moved from Jitterbug to Bugzilla as mentioned on the mailing list.

I don’t know the reason of the move, probably due to SPAM or interface improvements. If we look at those missing ids we can estimate the SPAM, note that some might be vulnerabilities on R (but I doubt that there are this much):

The first observable thing is the high number of ids missing on Jitterbug. I heard that this was an abuse of the site, which seems particularly bad arround two dates.

Later on when the system moved to Bugzilla there are much less missing ids, until one day there are around 120 missing ids and the date later to have an account it was required to send a message to the R core about it. Probably spamers abused the Bugzilla’s API. Is understandable that the R core team is concerned on receiving spam messages from those past experiences.

As the information from Jitterbug is both old and not as reach as those bugs on Bugzilla I will limit from now on to analyze those bugs reported on Bugzilla. In addition the number of bugs reported on Bugzilla is similar to the ones on Jitterbug:

Reported on	Reports
Jitterbug	3594
Bugzilla	3448

There are some old bugs opened on Jitterbug modified on Bugzilla (428) and still not closed.

Openning bugs reports

If we focus only on bugs reported on Bugzilla we find the following number of bugs:

Most bugs are closed followed by unconfirmed:

Status	Bug reports
CLOSED	2792
UNCONFIRMED	324
NEW	125
ASSIGNED	39
CONFIRMED	13
REOPENED	11
RESOLVED	9
VERIFIED	3

Do bugs reported have an attachment with a patch?

Attachment on opening	Patch	Bugs	%
No	No	380	43.18
No	Yes	500	56.82
Yes	No	438	61.60
Yes	Yes	273	38.40

Many bug reports have attachments on opening, mostly they contain code to reproduce the problem. If bugs do not contain attachments and they receive an attachment it will usually be a patch (but it might not).

What happens after submitting a bug report?

One of the most common thing to happen is that someone might comment on the bug either to ask for clarification or to discuss the bug report and possible solutions:

Most common action is receiving a comment, whose author is added on the CC field.

The R core is very active answering on the bug reports, only it seems that those that are trivial do not receive a reply as often as the other categories.

If we look by component and OS there are some combinations that received few comments, mostly those that are wishes for R.

If we split them between peitions to improve R and actual bug reports we can see a different pattern:

Enhancements usually receive less comments from the R core. Wishlists receive more comments from the R core among the enhancements.

What about the comments made by the original poster? Do they comment when they receive some feedback from other users?

Responsive	Bugs reports
no	409
yes	122

Most of them do not reply back when they receive a comment. There might be several causes, one of them is if their bug report is closed or assigned to an R core member.

handled	bugs
no	177
yes	232

Which shows that the majority of those who do not respond is because either the bug report is closed (fixed or not) or a R core member is assigned (usually to himself).

Who is active ?

So far we have explored the activity of users who report a bug and the R core members that receive this. But there are some users that beyond this they also participate and collaborate with the R core. To make it more recent we will look at those users active on the bugs opened on the last three years.

ID	Name	All comments	All attachments	Comments	Attachments	Bugs opened	Bugs interacted	Actions
3299	Elin Waring	100	2	99	1	1	66	101
963	Suharto Anggono	115	39	55	7	21	42	83
274	Sebastian Meyer	80	24	49	11	18	58	78
430	Benjamin Tyner	59	4	52	4	5	32	61
3256	Michael Chirico	110	51	2	1	52	54	55
1044	Kevin Ushey	65	10	4	0	35	39	39
1036	Henrik Bengtsson	56	14	9	1	26	31	36
2307	Lionel Henry	41	25	6	2	20	22	28
1056	Bill Dunlap	28	0	1	0	21	22	22
2801	Bob Rudis	19	4	16	2	2	12	20
1299	Gabriel Becker	52	28	0	0	18	18	18
11	Ben Bolker	14	0	11	0	3	9	14
114	Gabor Csardi	26	8	4	0	10	11	14
610	Jeroen Ooms	15	4	6	1	7	11	14
1602	brodie.gaslam@	53	16	0	0	13	13	13
6	Duncan Murdoch	22	7	0	0	12	12	12
921	Dirk Eddelbuettel	17	4	4	0	7	11	11
1715	Herv� Pag�s	24	1	5	1	5	7	11
2885	Jan Gorecki	14	1	1	0	9	10	10
3051	Xianying Tan	20	0	4	0	6	7	10
3228	Emil Bode	16	4	0	0	10	10	10
3344	Joe Cheng	7	3	7	3	0	1	10
2264	Neal Fultz	20	5	0	0	9	9	9
317	Mikko Korpela	9	0	5	0	3	6	8
847	Pavel N. Krivitsky	15	2	0	0	8	8	8
1251	Arni Magnusson	13	8	0	0	8	8	8
1849	Andre Mikulec	21	4	1	0	7	8	8
3330	André Gillibert	20	10	0	0	8	8	8
3376	Hangfan Zhang	14	3	0	0	8	8	8
2040	Bill Denney	16	1	0	0	7	7	7

(You can find a sortable anf filterable version of this table here). One of the top contributors was recently added as R core member. Other

Some contributors focus on providing patches, other’s open many bugs, other’s comment on the bugs to confirm bugs or provide context to the bugs.

Future of the report system ?

Last what is the progression of bugs

If we look only at bug reports and not enhancements every 1.52 day there is one bug report and one enhancement petitions per 4.44 days.

If we look only at bug reports and not enhancements there are 1.48 comments every day from R users, one comment on enhancement per 1.9 days.

Both of them has remained fairly constant over the years, at times a bit faster and sometimes slower.

Final comments

This is probably the post that took longer so far. I started on February to work on this blog post but until now I haven’t actually written the blog post I wanted.

Many thanks to Simon Urbanek for providing the database dump, without it would have been slower and harder if not impossible to do this analysis. Thanks to Heather Turner for encouraging me to do more on this project and providing valuable feedback about what kind of analysis could be useful and co-mentoring Pyush Kumar who I also like to thank the first analysis on the data and his contributions during GSoC. Many thanks to Gabe Becker and Michael Quirico for their feedback on the R contributors working group.

Now that the analysis is done, I want to finish the bugRzilla package (which I already started to use for small tasks on this analysis). I’m still working on it testing how is the best way to submit properly formatted bug reports with a developer instance set up by Simon. Then I’ll ask the R core if the way it submits bug reports works well for them.

TL:DR: There are many bug reported and handled by the R core and many users contribute on solving the bug reports. The peace of new bug reports and comments is constant as well as enhancements on the language itself.

Reproducibility

Session Info

## ─ Session info  ──────────────────────────────────────────────────────────────────────────────────────────────────────
##  hash: nail polish: medium skin tone, woman artist: dark skin tone, pager
## 
##  setting  value
##  version  R version 4.1.2 (2021-11-01)
##  os       Ubuntu 20.04.3 LTS
##  system   x86_64, linux-gnu
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       Europe/Madrid
##  date     2021-11-16
##  pandoc   2.14.0.3 @ /usr/lib/rstudio/bin/pandoc/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version   date (UTC) lib source
##  assertthat    0.2.1     2019-03-21 [1] CRAN (R 4.1.2)
##  backports     1.3.0     2021-10-27 [1] CRAN (R 4.1.2)
##  bit           4.0.4     2020-08-04 [1] CRAN (R 4.1.2)
##  bit64         4.0.5     2020-08-30 [1] CRAN (R 4.1.2)
##  blob          1.2.2     2021-07-23 [1] CRAN (R 4.1.2)
##  blogdown      1.6       2021-11-09 [1] CRAN (R 4.1.2)
##  bookdown      0.24      2021-09-02 [1] CRAN (R 4.1.2)
##  broom         0.7.10    2021-10-31 [1] CRAN (R 4.1.2)
##  bslib         0.3.1     2021-10-06 [1] CRAN (R 4.1.2)
##  bugRzilla   * 0.0.90001 2021-11-13 [1] Github (llrs/bugRzilla@24bc5de)
##  cachem        1.0.6     2021-08-19 [1] CRAN (R 4.1.2)
##  cli           3.1.0     2021-10-27 [1] CRAN (R 4.1.2)
##  colorspace    2.0-2     2021-06-24 [1] CRAN (R 4.1.2)
##  crayon        1.4.2     2021-10-29 [1] CRAN (R 4.1.2)
##  curl          4.3.2     2021-06-23 [1] CRAN (R 4.1.2)
##  DBI         * 1.1.1     2021-01-15 [1] CRAN (R 4.1.2)
##  dbplyr      * 2.1.1     2021-04-06 [1] CRAN (R 4.1.2)
##  digest        0.6.28    2021-09-23 [1] CRAN (R 4.1.2)
##  dplyr       * 1.0.7     2021-06-18 [1] CRAN (R 4.1.2)
##  ellipsis      0.3.2     2021-04-29 [1] CRAN (R 4.1.2)
##  evaluate      0.14      2019-05-28 [1] CRAN (R 4.1.2)
##  fansi         0.5.0     2021-05-25 [1] CRAN (R 4.1.2)
##  farver        2.1.0     2021-02-28 [1] CRAN (R 4.1.2)
##  fastmap       1.1.0     2021-01-25 [1] CRAN (R 4.1.2)
##  forcats     * 0.5.1     2021-01-27 [1] CRAN (R 4.1.2)
##  generics      0.1.1     2021-10-25 [1] CRAN (R 4.1.2)
##  ggpattern   * 0.2.2     2021-11-11 [1] Github (coolbutuseless/ggpattern@7214181)
##  ggplot2     * 3.3.5     2021-06-25 [1] CRAN (R 4.1.2)
##  ggrepel     * 0.9.1     2021-01-15 [1] CRAN (R 4.1.2)
##  glue          1.5.0     2021-11-07 [1] CRAN (R 4.1.2)
##  gtable        0.3.0     2019-03-25 [1] CRAN (R 4.1.2)
##  highr         0.9       2021-04-16 [1] CRAN (R 4.1.2)
##  htmltools     0.5.2     2021-08-25 [1] CRAN (R 4.1.2)
##  httr          1.4.2     2020-07-20 [1] CRAN (R 4.1.2)
##  jquerylib     0.1.4     2021-04-26 [1] CRAN (R 4.1.2)
##  jsonlite      1.7.2     2020-12-09 [1] CRAN (R 4.1.2)
##  kableExtra    1.3.4     2021-02-20 [1] CRAN (R 4.1.2)
##  knitr         1.36      2021-09-29 [1] CRAN (R 4.1.2)
##  labeling      0.4.2     2020-10-20 [1] CRAN (R 4.1.2)
##  lattice       0.20-45   2021-09-22 [1] CRAN (R 4.1.2)
##  lifecycle     1.0.1     2021-09-24 [1] CRAN (R 4.1.2)
##  lubridate   * 1.8.0     2021-10-07 [1] CRAN (R 4.1.2)
##  magrittr      2.0.1     2020-11-17 [1] CRAN (R 4.1.2)
##  Matrix        1.3-4     2021-06-01 [1] CRAN (R 4.1.2)
##  memoise       2.0.0     2021-01-26 [1] CRAN (R 4.1.2)
##  mgcv          1.8-38    2021-10-06 [1] CRAN (R 4.1.2)
##  munsell       0.5.0     2018-06-12 [1] CRAN (R 4.1.2)
##  nlme          3.1-153   2021-09-07 [1] CRAN (R 4.1.2)
##  patchwork   * 1.1.1     2020-12-17 [1] CRAN (R 4.1.2)
##  pillar        1.6.4     2021-10-18 [1] CRAN (R 4.1.2)
##  pkgconfig     2.0.3     2019-09-22 [1] CRAN (R 4.1.2)
##  purrr         0.3.4     2020-04-17 [1] CRAN (R 4.1.2)
##  R6            2.5.1     2021-08-19 [1] CRAN (R 4.1.2)
##  Rcpp          1.0.7     2021-07-07 [1] CRAN (R 4.1.2)
##  rlang         0.4.12    2021-10-18 [1] CRAN (R 4.1.2)
##  rmarkdown     2.11      2021-09-14 [1] CRAN (R 4.1.2)
##  RMySQL      * 0.10.22   2021-06-22 [1] CRAN (R 4.1.2)
##  RSQLite     * 2.2.8     2021-08-21 [1] CRAN (R 4.1.2)
##  rstudioapi    0.13      2020-11-12 [1] CRAN (R 4.1.2)
##  rvest         1.0.2     2021-10-16 [1] CRAN (R 4.1.2)
##  sass          0.4.0     2021-05-12 [1] CRAN (R 4.1.2)
##  scales        1.1.1     2020-05-11 [1] CRAN (R 4.1.2)
##  sessioninfo   1.2.1     2021-11-02 [1] CRAN (R 4.1.2)
##  stringi       1.7.5     2021-10-04 [1] CRAN (R 4.1.2)
##  stringr       1.4.0     2019-02-10 [1] CRAN (R 4.1.2)
##  svglite       2.0.0     2021-02-20 [1] CRAN (R 4.1.2)
##  systemfonts   1.0.3     2021-10-13 [1] CRAN (R 4.1.2)
##  tibble        3.1.6     2021-11-07 [1] CRAN (R 4.1.2)
##  tidyr         1.1.4     2021-09-27 [1] CRAN (R 4.1.2)
##  tidyselect    1.1.1     2021-04-30 [1] CRAN (R 4.1.2)
##  utf8          1.2.2     2021-07-24 [1] CRAN (R 4.1.2)
##  vctrs         0.3.8     2021-04-29 [1] CRAN (R 4.1.2)
##  viridisLite   0.4.0     2021-04-13 [1] CRAN (R 4.1.2)
##  webshot       0.5.2     2019-11-22 [1] CRAN (R 4.1.2)
##  withr         2.4.2     2021-04-18 [1] CRAN (R 4.1.2)
##  xfun          0.28      2021-11-04 [1] CRAN (R 4.1.2)
##  xml2          1.3.2     2020-04-23 [1] CRAN (R 4.1.2)
##  yaml          2.2.1     2020-02-01 [1] CRAN (R 4.1.2)
## 
##  [1] /home/lluis/bin/R/4.1.2/lib/R/library
## 
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

A program from Google where they sponsor students and organizations to work on open source projects. I didn’t knew R participated, but here is the organization on Github.↩︎

Social activities on GitHub

Sun, 21 Jun 2020 00:00:00 +0000

On my last post I explored the Bioconductor submissions using {gh} to retrieve some data. After some feedback from the Bioconductor community I realized I should download other kind of data to improve my analysis on the reviews.

To make this I developed a new package to retrieve information from GitHub.

socialGH

This package based on {gh}, allows to retrieve, data from Github.

You can install it with

remotes::install_github("llrs/socialGH")

Basically pulls the data in list format and transforms it into a data.frame in order to be able to do analysis, filter it or analyze it.

library("socialGH")
library("tidyverse")
## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
## ✓ ggplot2 3.3.3     ✓ purrr   0.3.4
## ✓ tibble  3.0.4     ✓ dplyr   1.0.2
## ✓ tidyr   1.1.2     ✓ stringr 1.4.0
## ✓ readr   1.4.0     ✓ forcats 0.5.0
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

It allows to selective download comments, pull requests, issues, events, labels and the timeline of an issue.

With the issues we can see the labels, how many coments and many information:

issues_blog <- get_issues("llrs/blogR")
dim(issues_blog)
## [1] 46 15
colnames(issues_blog)
##  [1] "assignees"   "assignee"    "label"       "state"       "locked"     
##  [6] "milestone"   "n_comments"  "title"       "created"     "updated"    
## [11] "association" "text"        "id"          "closer"      "poster"
# Labels used
issues_blog %>% 
  pull(label) %>% 
  unlist(FALSE, FALSE) %>% 
  table()
## .
##  b101nfo.blogspot.com          Bioconductor                config 
##                     1                     4                     1 
##                  CRAN             goverment           help wanted 
##                     3                     1                     1 
##               invalid               package                  Post 
##                     1                     2                    27 
##              question              rOpenSci todo 🗒 
##                     2                     1                     4 
##               website 
##                     1
count(issues_blog, state)
##    state  n
## 1 closed 27
## 2   open 19
count(issues_blog, n_comments)
##   n_comments  n
## 1          0 24
## 2          1 15
## 3          2  2
## 4          3  4
## 5          5  1

However, it doesn’t retrieve each comment of an issue.

# Issues with comments
issues_blog %>% 
  filter(n_comments > 0) %>% 
  pull(id)
##  [1] 46 42 40 39 36 34 33 29 28 26 25 23 16 10  9  8  7  6  5  4  3  2

comments <- get_comments("llrs/blogR")
dim(comments)
## [1] 36  6
colnames(comments)
## [1] "text"        "created"     "updated"     "association" "id"         
## [6] "commenter"
count(comments, association)
##   association  n
## 1       OWNER 36

We can see that I was the only one writing on the issues and we already retrieved the text of the comments.

We can also look for events on issues:

events <- get_events("llrs/blogR", 23)
count(events, event)
##     event n
## 1 labeled 1
## 2  closed 1

On all the functions you can provide a number of the issue and you’ll retrieve the information just for that issue. If you don’t provide an issue it will search the whole repository:

events <- get_events("llrs/blogR")
count(events, event)
##                 event  n
## 1             labeled 50
## 2              closed 27
## 3             renamed  4
## 4 marked_as_duplicate  1
## 5            assigned  6
## 6          subscribed  3
## 7           mentioned  3
## 8           unlabeled  1

However it is better if we look to the timeline of an issue:, which downloads each comment of the issues.

gt <- get_timelines("llrs/blogR", 23)
## Warning: This is under preview and may fail.
gt[, c("label", "event", "created", "association", "actor")]
##   label     event             created association             actor
## 1    NA commented 2020-02-14 00:39:47       OWNER llrs, User, FALSE
## 2    NA commented 2020-02-14 09:44:26       OWNER llrs, User, FALSE
## 3  Post   labeled 2020-02-18 10:10:35        <NA> llrs, User, FALSE
## 4    NA commented 2020-02-29 17:58:51       OWNER llrs, User, FALSE
## 5    NA    closed 2020-02-29 17:58:51        <NA> llrs, User, FALSE

With timeline we don’t get the initial information of when the issue was created and we’ll need to call get_issue("llrs/blogR", 23) to know that. Here I did omit the text of the comment to make it readable, but we can see what has been happening and by who or who is affecting.

Learning

Developing this package I learned more about the {gh} package (In the previous blog I wrote manually the calls to different pages, which later on I discovered it is automatically handled by {gh}). And learned that the different accept headers have influenced on the total information returned (and that you cannot pass several accept headers at the same time).
Hope to learn more about the R community that is using Github as a way to help each other, improve packages and process.

Reproducibility

## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 4.0.1 (2020-06-06)
##  os       Ubuntu 20.04.1 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       Europe/Madrid               
##  date     2021-01-08                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
##  package     * version date       lib source                           
##  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.1)                   
##  backports     1.2.1   2020-12-09 [1] CRAN (R 4.0.1)                   
##  blogdown      0.21.84 2021-01-07 [1] Github (rstudio/blogdown@c4fbb58)
##  bookdown      0.21    2020-10-13 [1] CRAN (R 4.0.1)                   
##  broom         0.7.3   2020-12-16 [1] CRAN (R 4.0.1)                   
##  cellranger    1.1.0   2016-07-27 [1] CRAN (R 4.0.1)                   
##  cli           2.2.0   2020-11-20 [1] CRAN (R 4.0.1)                   
##  colorspace    2.0-0   2020-11-11 [1] CRAN (R 4.0.1)                   
##  crayon        1.3.4   2017-09-16 [1] CRAN (R 4.0.1)                   
##  curl          4.3     2019-12-02 [1] CRAN (R 4.0.1)                   
##  DBI           1.1.0   2019-12-15 [1] CRAN (R 4.0.1)                   
##  dbplyr        2.0.0   2020-11-03 [1] CRAN (R 4.0.1)                   
##  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.1)                   
##  dplyr       * 1.0.2   2020-08-18 [1] CRAN (R 4.0.1)                   
##  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.1)                   
##  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.1)                   
##  fansi         0.4.1   2020-01-08 [1] CRAN (R 4.0.1)                   
##  forcats     * 0.5.0   2020-03-01 [1] CRAN (R 4.0.1)                   
##  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.1)                   
##  generics      0.1.0   2020-10-31 [1] CRAN (R 4.0.1)                   
##  ggplot2     * 3.3.3   2020-12-30 [1] CRAN (R 4.0.1)                   
##  gh            1.2.0   2020-11-27 [1] CRAN (R 4.0.1)                   
##  gitcreds      0.1.1   2020-12-04 [1] CRAN (R 4.0.1)                   
##  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  gtable        0.3.0   2019-03-25 [1] CRAN (R 4.0.1)                   
##  haven         2.3.1   2020-06-01 [1] CRAN (R 4.0.1)                   
##  hms           0.5.3   2020-01-08 [1] CRAN (R 4.0.1)                   
##  htmltools     0.5.0   2020-06-16 [1] CRAN (R 4.0.1)                   
##  httr          1.4.2   2020-07-20 [1] CRAN (R 4.0.1)                   
##  jsonlite      1.7.2   2020-12-09 [1] CRAN (R 4.0.1)                   
##  knitr         1.30    2020-09-22 [1] CRAN (R 4.0.1)                   
##  lifecycle     0.2.0   2020-03-06 [1] CRAN (R 4.0.1)                   
##  lubridate     1.7.9.2 2020-11-13 [1] CRAN (R 4.0.1)                   
##  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.1)                   
##  modelr        0.1.8   2020-05-19 [1] CRAN (R 4.0.1)                   
##  munsell       0.5.0   2018-06-12 [1] CRAN (R 4.0.1)                   
##  pillar        1.4.7   2020-11-20 [1] CRAN (R 4.0.1)                   
##  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.1)                   
##  purrr       * 0.3.4   2020-04-17 [1] CRAN (R 4.0.1)                   
##  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.1)                   
##  Rcpp          1.0.5   2020-07-06 [1] CRAN (R 4.0.1)                   
##  readr       * 1.4.0   2020-10-05 [1] CRAN (R 4.0.1)                   
##  readxl        1.3.1   2019-03-13 [1] CRAN (R 4.0.1)                   
##  reprex        0.3.0   2019-05-16 [1] CRAN (R 4.0.1)                   
##  rlang         0.4.10  2020-12-30 [1] CRAN (R 4.0.1)                   
##  rmarkdown     2.6     2020-12-14 [1] CRAN (R 4.0.1)                   
##  rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.0.1)                   
##  rvest         0.3.6   2020-07-25 [1] CRAN (R 4.0.1)                   
##  scales        1.1.1   2020-05-11 [1] CRAN (R 4.0.1)                   
##  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.1)                   
##  socialGH    * 0.0.3   2020-08-17 [1] local                            
##  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.1)                   
##  stringr     * 1.4.0   2019-02-10 [1] CRAN (R 4.0.1)                   
##  tibble      * 3.0.4   2020-10-12 [1] CRAN (R 4.0.1)                   
##  tidyr       * 1.1.2   2020-08-27 [1] CRAN (R 4.0.1)                   
##  tidyselect    1.1.0   2020-05-11 [1] CRAN (R 4.0.1)                   
##  tidyverse   * 1.3.0   2019-11-21 [1] CRAN (R 4.0.1)                   
##  vctrs         0.3.6   2020-12-17 [1] CRAN (R 4.0.1)                   
##  withr         2.3.0   2020-09-22 [1] CRAN (R 4.0.1)                   
##  xfun          0.20    2021-01-06 [1] CRAN (R 4.0.1)                   
##  xml2          1.3.2   2020-04-23 [1] CRAN (R 4.0.1)                   
##  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.1)                   
## 
## [1] /home/lluis/bin/R/4.0.1/lib/R/library