Practical Reproducibility Instruction: Teaching Reproducibility with Technical and Non-Technical Tools

10/13/2021

Slides Usage Notes

Recommended to press w for widescreen view.
Also recommended to press f for fullscreen view.
Press p for Presenter Notes.
These slides are licensed as CC-BY.

About UVA and UVA Library

University of Virginia:

R1 university
23,000 students and 12 Schools
Only the School of Data Science has a policy for open access and open data

UVA Library:

Our team members come from Scholarly Communications and Research Data Services & Social, Natural, and Engineering Sciences, which are teams from different units in the library.
Research Data Services has a long-running workshop series.
It is very rare for library staff to teach for-credit courses at UVA.

JENN: 1 min (spend 1 minute here)

Welcome to our talk today about how we implemented a Reproducibility workshop series at UVA Library. I am Jennifer Huck; I’m the Data Librarian and Acting Associate Director of Research Data Services at UVA Library. I am joined by Sherry Lake, the Scholarly Repository Librarian, and Erich Purpur, a Science and Engineering Research Librarian.

Here is a little about the University of Virginia:

R1 university.
There are about 23,000 students and there are 12 Schools.
Only the School of Data Science has a policy for open access and open data

And some context for UVA Library:

Our team members come from Scholarly Communications and Research Data Services & Social, Natural, and Engineering Sciences, which are teams from different units in the library.
Research Data Services has a long-running workshop series on tools for quantitative and qualitative data analysis.
One thing to point out: It is very rare for library staff to teach for-credit courses at UVA. Teaching a within a workshop series like this is one of our best chances for instruction.

What We Did

A new workshop series: ‘Reproducible Research for Early Graduate Students’

Included 2 technical and 2 non-technical sessions:

Technical sessions: tools and workflows
Non-Technical sessions: conceptual instruction on organization and data sharing

How We Did It

We launched it as a standalone mini-series as part of Research Data Service’s ongoing workshop series
We leveraged a relatively varied skillset among library staff to present these sessions.
Sessions were a mix of old and new workshop content
Outreach: Research Data Services Newsletter, reaching out to communications directors at various schools
Virtual (Spring 2021) and In Person (Fall 2021)

JENN: 4 min (spend 1 min here)

Each semester, our Research Data Services team presents an extensive suite of workshops taught by our staff, as well as collaborators from other units throughout the university. We offer training on technical tools, such as R, Python, Tableau, and Dedoose. We launched our Reproducibility series as a standalone mini-series within that workshop environment.
We are fortunate to have a lot of staff at our library with some facility in data. We were able to pull in librarians with varied skillsets.
Our sessions were a mix of old and new workshop content. Of the sessions in our reproducibility series, one was an existing workshop, one was a mashup of old and new content, and two were brand new.
As for Outreach, we advertised the sessions in our Research Data Services Newsletter which goes out to interested readers at UVA, and we also reached out directly to communications directors at various schools who then send info to their students.
We had two presentation formats. This Spring, we presented virtually via Zoom, and this Fall we presented in person.

Workshop Details:
Non-Technical Sessions

Organizing Files and Metadata for Transparent & Reproducible Research

Project organization: demonstrated & discussed the concepts
- Organizing files: directory (grouping) structure
- Documentation (e.g., READMEs)
- File Naming

SHERRY: 5 min (spend 1 min here)

The workshop series started with the non-technical workshop: Organizing Files and Metadata for Transparent & Reproducible Research

In this workshop, participants were instructed on fundamental approaches on creating a research compendium, which is the foundation of transparent and reproducible research.

Participants learned what kinds of documents and materials they should create and preserve; what type of information the documents should contain; and how they should be formatted and organized. Topics for this workshop included organizing raw data (files), data analysis, documentation, scripts, metadata, readme files, project organization, and naming conventions.

The examples were based in R, but no programming experience or experience with R was required, as the information learned in the workshop could be applied to any quantitative programming environment.

Sharing Your Data for Transparent and Reproducible Research

Workshop Details:
Technical Sessions

Version Control with Git and GitHub

Learn about version control
Install git and make Github account
Common workflow scenarios
- local changes
- push to remote repository
- branching/collaboration

RStudio and R Markdown

Review RStudio Projects to create a project-oriented workflow in R.
Overview of R Markdown for literate programming, including new features of Visual R Markdown.

How You Can Do It

Look for partners at your institution to partner with
You don’t need to be an expert - skills can be emerging
Re-brand existing workshops and classes at your library
Take our stuff!

JENN: 10 min (spend 2 min here)

Here are some takeaways:

Look for partners to collaborate on a series of reproducibility. One person does not and should not have to do it all! We partner with other library colleagues outside of Research Data Services and units outside of the library. For example, we collaborate with librarians like Sherry, who is not a part of Research Data Services and also with our Health Sciences Library and Research Computing colleagues to offer a wide variety of workshops.
We all have various levels of technical expertise. Some technical experience is definitely helpful but you don’t have to be a data scientist to teach in this area. I consider my skills in R to be “emerging,” but I found that that was enough for me to “see” the value in tools (especially in RStudio) or concepts that might be overlooked or not formally taught. Not all these tools require a strong statistical, data science, or programming background in order to teach them.
Look around for existing workshops at your library to market or re-brand to fit into a Reproducibility theme. Are you already doing things you could use for this purpose? Sherry will expand on this idea in the next slide.
Finally, all of our workshop material is openly available. Take our stuff! There is no need to create all the materials from scratch. Note that the version control workshop is a combination of previous workshop materials plus online content that created externally.

Transform Data Management Workshops

(Jones 2018)

SHERRY: 12 min (spend 2 min here)

I want to spend a couple of minutes making a link from typical data management and sharing workshops to transparency and reproducibility.

Doing research for transparency and managing data (data management) [and FAIR] are overlapping, but distinct concepts each emphasizing different aspects of handling and sharing research data.

Knowing about these “differences” (and strengths) can help librarians promote good research practices. Sharing - promote access & reuse, DM is basis from the start of a project in order for sharing to be meaningful.

Research Data Management (RDM): is not a goal in itself, but a set of practices to handle information collected and created during research. RDM is the compilation of many small practices that make your data easier to find, easier to understand, less likely to be lost, and more likely to be usable during a project or ten years later.

FAIR principles advocate for increased findability, accessibility, interoperability and reusability of research data and scholarly digital objects more generally.

FAIR and transparency are “noble” aspirations & useful way to engage researchers and encourage good data practices from the outset. FAIR/Reproducibility are more catchy concepts to engage researchers in DM.

FAIR and transparency extends DM - data & research is not just managed but are FAIR and can be understood, reproducible. Data that is not “managed” cannot be successfully shared/re-used (FAIR).

Thus… we can’t forget data management - DM is needed before data can be FAIR and shared - it enables data to be created and used in a way that increases transparency helps in sharing and reuse. BUT…. maybe by using the more appealing language of FAIR and organizing research, we can engage researchers in data management too. Note: Jenn did not use the phrase “Data Management” at all in her workshop/title or presentation, BUT file organization, file naming, documentation are all basic DM 101.

FAIR and transparency both focus on data sharing, ensuring content is made available in ways that promote access and reuse. Data management by contrast is about the stewardship of data from the point of conception onwards. It makes no assumptions about access, but is essential if data are to be meaningful to others.

The way we presented the concepts of reproducible research & sneaking in DM concepts (without calling them that) and FAIR practices is a useful way to engage researchers and encourage good data practices from the outset. It might open conversations with researchers and funders in ways that dull old data management never did.

Research Data Management is the foundation for sharing. Managing and sharing research data are not a high priority for researchers - choosing the right “terminology” (concepts of FAIR, organization, sharing, transparency, reproducibility) can help in the up-take of best practices for managing and sharing.

Resources To Start With

Practical Reproducibility
- Transparent and Reproducible Social Science Research (Christensen 2019)
- Inspiration for a multi-part course on Reproducibility: Reproducible Research Techniques for Synthesis: (National Center for Ecological Analysis and Synthesis, n.d.)
Organize for Transparent and Reproducible Research
- Reproducible Science Curriculum: Data & Project Organization (Data Carpentry, n.d.)
- How to Name Files (Bryan 2015)
Version Control with Git and GitHub
- Git Command Line Basics [https://www.youtube.com/watch?v=HVsySz-h9r4]
Sharing Your Data for Transparent and Reproducible Research
- Data Management vs FAIR (Higman, Bangert, and Jones 2019)
- Best Practices (Briney, Coates, and Goben 2020)
Reproducible Analysis and Documentation in RStudio and R Markdown
- Reproducible Research With R and RStudio (Gandrud 2020)

Feedback and Assessment

Track attendance
Post workshop follow up

Looking Forward

Some ideas:

Do specific outreach to departments on this kind of training
Find a seminar series that this could fit into
Find “reproducibility champions” who want their grad student to learn these concepts

More Resources

Our Workshop materials are available at OSF: https://osf.io/ws3u2
- There are considerably more references to learn more about each individual topic in the workshop slides
Research Data Management Services Zotero group library:
https://www.zotero.org/groups/361046/research_data_management_services/library

References

Briney, Kristin, Heather Coates, and Abigail Goben. 2020. “Foundational Practices of Research Data Management.” Research Ideas and Outcomes 6 (July): e56508. https://doi.org/10.3897/rio.6.e56508.

Bryan, Jenny. 2015. “How to Name Files.” Speaker Deck. https://speakerdeck.com/jennybc/how-to-name-files.

Christensen. 2019. Transparent and Reproducible Social Science Research. First edition. Oakland, California: University of California Press.

Data Carpentry. n.d. “Reproducible Science Curriculum: Data & Project Organization.” https://reproducible-science-curriculum.github.io/organization-RR-Jupyter/.

Gandrud, Christopher. 2020. Reproducible Research with R and RStudio. 3rd edition. Boca Raton, FL: Chapman and Hall/CRC.

Higman, Rosie, Daniel Bangert, and Sarah Jones. 2019. “Three Camps, One Destination: The Intersections of Research Data Management, FAIR and Open.” Insights 32 (1): 18. https://doi.org/10.1629/uksg.468.

Jones, Sarah. 2018. “Open Data, FAIR Data and RDM: The Ugly Duckling.” Berlin. https://doi.org/10.5281/zenodo.1196631.

National Center for Ecological Analysis and Synthesis. n.d. “Reproducible Research Techniques for Synthesis.” National Center for Ecological Analysis and Synthesis. https://www.nceas.ucsb.edu/learning-hub/short-course.