Exploring Data Science Education: From Tutorials to Assessment

Duke Statistical Science | Graduation with Distinction

Evan Dragich

supervised by Mine Çetinkaya-Rundel, PhD.

April 11, 2023

About Me

Statistical Science B.S. & Psychology B.S.
Combination of previous experiences and interest
- STEM education research
- Started StatSci sophomore spring
- TAing Intro Data Science (STA199)

I’m a senior double majoring in statsci and Psych. I had a bit of a roundabout path to the statsci major: I started college premed, wasn’t entirely sure what I was doing, but knew I enjoyed STEM and quantitative problem solving. Got involved in research, working on a STEM education project that was possibly slightly out of my depth at the time and required a lot of teaching myself statistics. Realizing I wasn’t as interested in the medicine aspects of medicine, more the problem-solving aspects of research, enjoyed statistical aspects more than I thought I would, so made somewhat of a leap of faith and declared statsci major. Due to making this decision so late, and having solid baseline from Psych major stat classes, AP stat in high school, research experiences, foregoing 199 (intro data science) to unlock higher-level electives earlier. Next semester, became TA for STA 199. Enjoyed thinking about the curriculum and how things were presented, also knew that the Statsci major undergoing some overhaul during my time. Knew I wanted to do an independent study and possibly a thesis at that point, asked Prof. Çetinkaya-Rundel if she had any research projects I could work on, and the rest was history.

Thesis TOC/Agenda

Thesis divided into 2 strands:
- Building a introductory data science concept inventory-style assessment
- Building dsbox, an introductory data science tutorial package
Agenda
- Background
- Initial Steps
- Interview Process
- Item Case Studies
- Package Construction + Examples
- Discussion
- Q&A

My overall thesis project consists of two separate, yet related, projects. First, the construction of a concept inventory-style assessment to measure students’ learning outcomes in introductory data science classes. Secondly, continuing the themes of introductory data science education and scalability, the construction and release of a package, dsbox, which contains 10 standalone tutorials testing similar concepts. those are the two things that are both helpful (they are busy developing own course content), developing tools to help teachers ASSESS their teaching, and the other is to help teachers SCALE teaching

The structure of my presentation today will be as follows: I will first motivate the need for such an assessment and the process used to develop it. Then, I will quickly discuss the initial steps of my work on the assessment before describing the interview process and taking time to dive into some case studies of several items which form a representative sample of the general iterative process of item creation and refinement. Then, I’ll shift gears to give an overview of the dsbox package’s goals, structure, and examples from tutorials. Finally, I’ll conclude with a reflective discussion before leaving time for questions at the end.

Building a Data Science Assessment

Background

Concept inventories for educational research
- CAOS for statistics
Data science (DS) as it emerges as a field–what is it, exactly?
How exactly do people: (1) make, (2) pilot, (3) validate new concept inventories or scales?

Say you’re an xyz professor and you want to do research–will flipping the lectures and lab of your classroom lead to better learning outcomes, do earlier lab sections perform better than afternoon ones, etc. What do you use to measure? some sort of assessment. Good thing a standardized one exists for calculus, etc. This assessment is most effective if its standard across courses/“validated’, gives legitimacy to what you do in class, helpful to use as a legit benchmark for your teaching compared to random self-contained assessment. Work on such an assessment for introductory statistics, CAOS, led to paper from 2007 has 418 citations! Clearly this is something people are interested in. We look now to expand that to introductory data science courses–but first, what exactly is data science? As more and more courses and majors emerge, reviews of syllabi have found heavy focuses on computing or pseudocode, data wrangling and visualization, with some focus on statistics, modeling, as well as some smaller ethics component. Once a scope has been set and questions drafted, how do we prepare such an assessment for a pilot? Key findings from research on think-aloud interviews has found that testing a variety of questions with specific misconceptions in mind via think-aloud interviews, in which a target population is asked to reason out loud through their thought process. After making refinements based on students’ perceptions, assessments are then given at a large scale to collect enough data to test possible subscales. For example, if students’ performance on questions on similar topics seemed to be correlated, there is a potential to calculate a separate subscore on that topic, such as data visualization. Those steps of the process constitute my work on the assessment portion of the thesis, and now I will now dive deeper into each phase of the development.

Initial cleaning

Combine questions into single set of passages and items
Draft into Quarto Book for easy browsing

Initial cleaning

Interviews

Two rounds of interviews:
- 3 faculty
- 3 intro DS teaching assistants

Interviews: Faculty

Flow:
- What topics must be in an introductory data science course?
- What topics are nice to have in an introductory data science course?
- Think-aloud thought process
- Additional comments or suggestions
- What are the strengths of the current assessment?
- What topics are missing from the current assessment?
- What is in the current assessment, but doesn’t belong?
Themes from faculty interviews
- CS vs. Statistics perspectives
- Context concerns
- Cognitive load

Interviews: Students

Flow:
- Think-aloud thought process
- Additional comments or suggestions
- Are the pacing and length appropriate?
- Based on what you remember learning in intro data science, what topics are missing from the current assessment?
- Based on what you remember learning in intro data science, what is in the current assessment, but doesn’t belong?
Themes from student interviews
- General agreement
- Gradient of mastery

Current Prototype

15 passages, 26 items

Passage	Learning Objective(s)
Storm Paths	modeling; simulation; uncertainty
Movie Budgets 1	compare summary statistics visually
Movie Budgets 2	modeling; $R^2$; compare trends visually
Application Screening	ethics; modeling; proxy variable
Banana Conclusions	causation; statistical communication
COVID Map	complex visualization; spatial data; time series; sophisticated scales
He Said She Said	basic visualization; sophisticated scales
Build-a-Plot	data to visualization process
Disease Screening	compare classification diagnostics visually
Realty Tree	modeling; regression tree; variable selection
Website Testing	compare trends visually; uncertainty; modeling; time series; extrapolation
Image Recognition	ethics; modeling; representativeness of training data
Data Confidentiality	ethics; data deidentification; statistical communication
Activity Journal	structure data; store data
Movie Wrangling	data cleaning; data wrangling; column-wise string operations; pseudocode; joins

Case Study: Application Screening

You are working on a team that is making a deterministic model to quickly screen through applications for a new position at the company. Based on employment laws, your model may not include variables such as age, race, and gender, which could be potentially discriminatory.

Your colleague suggests including a rule that eliminates candidates with more than 20 years of previous work experience, because they may have high salary expectations. Why might using this variable be considered unethical? Explain your answer.

Case Study: Application Screening

Your colleague suggests including a rule that eliminates candidates with more than 20 years of previous work experience, because they may have high salary expectations. Are there ethical implications of using this variable to select candidates? Explain your answer.

Well… that doesn’t help much. We still have the classic selection bias, or the “presence vs absence” heuristic clouding results; students would think “well, if there wasn’t an ethical problem, they wouldn’t have included this as one of the only ethics questions on the assessment.” Plus, how are we grading this? What are we looking for to confirm that they understand the proxy variable? It might work to set up an autograder that marks “correct” if they mark “yes” to the ethics question AND mention “proxy” in their response. But, this is an introductory-level assessment. Will students be able to concisely describe employment experience as a “proxy,” or would explanations be wordier and might include “is correlated with,” “is related to,” “goes hand in hand with,” “predicts.” If we want auto gradable, looking like MC is going to be main way to go. Note that, at this stage, Application Screening and several other similar questions are left in open-ended format to collect this type of data from the pilots.

Case Study: Data Confidentiality

A newspaper reports on the results of a survey from a small (<2000 student) college. The college agrees to have the data released to the public so long as the students’ identities and academic standing information are kept confidential. Which of the following combinations of variables is less likely to unintentionally identify any students? Explain.

a. Year, major, sports played

b. Year, major

Case Study: Data Confidentiality

A newspaper reports on the results of a survey from a small (<2000 student) university. The university agrees to have the data released to the public so long as the students’ identities and academic standing information are kept confidential. Which of the following combinations of variables is less likely to unintentionally identify any students? Explain.

a. Year, major, sports played

b. Year, major

Case Study: Data Confidentiality

A newspaper reports on the results of a survey from a small (<2000 student) university. The university agrees to have the data released to the public so long as the students’ identities and academic standing information are kept confidential. Which of the following combinations of variables is less likely to unintentionally identify any students? Explain.

a. Class year and sports played

b. Student ID and dorm zip code

c. GPA and major

d. Birth date and phone number

e. None of the above

I really like this current version for a few reasons. First, “none of the above.” It doesn’t solve everything, but the closest you can get to having an open-ended prompt in a true MC assignment is to have a “none of the above,” and to use it! (roughly proportionally, so ~20-25% of the time). Second, the options are well-distributed and take some time to think through. We refined this in the gap between faculty and student interviews, and all three students had a similar response. A nice rhythm as they read through the responses, respectively “hmm, definitely not, ummmm maybe?, yeah definitely not, uh”. Immediately eliminating “b” and “d”, students debated “a” and “c” on a metric we hadn’t even surmised when writing. “Well, in terms of figuring out who people are, class year and sports played would do that. BUT, knowing GPA and major wouldn’t really let you identify anyone, but GPA is more sensitive data than class year/sports played in that it would be more harmful if released, and linked back to the people.” Out of all the think-aloud interviews, this single moment stood out as justifying why we asked students to walk through their reasoning processes, and to “think out loud:” to catch unexpected thought processes like this.

Case Study: Movie Budgets 1

A data scientist at IMDb has been given a dataset comprised of the revenues and budgets for 2,349 movies made between 1986 and 2016.

Suppose they want to compare several distributional features of the budgets among four different genres—Horror, Drama, Action, and Animation. To do this, they create the following plots.

Case Study: Movie Budgets 1

Fill in the following table by placing a check mark in the cells corresponding to the attributes of the data that can be determined by examining each of the plots.

	Plot A	Plot B	Plot C	Plot D
Mean	☐	☐	☐	☐
Median	☐	☐	☐	☐
IQR	☐	☐	☐	☐
Shape	☐	☐	☐	☐

Case Study: Movie Wrangling

The table below provides data about 10 movies released in the United States. It provides data on the movie’s title , the movie’s director, the date the movie was released, the season the movie was released, the worldwide gross intake in U.S. dollars, the cleaned version of the worldwide gross intake in U.S. dollars, and whether or not the movie won the Best Picture Oscar.

Case Study: Movie Wrangling

Movies Table
title	director	release_date	season	gross	gross_clean	best_picture
Almost Famous	Cameron Crowe	22 September 2000	Fall	$47.39M	47.39	No
CODA	Sian Heder	13 August 2021	Summer	$1.61M	1.61	Yes
E.T. the Extra-Terrestrial	Steven Spielberg	11 June 1982	Summer	$792.91M	792.91	No
Luca	Enrico Casarosa	18 June 2021	Summer	$49.75M	49.75	No
Middle of Nowhere	Ava DuVernay	1 September 2014	Fall	$0.24M	0.24	No
Moonlight	Barry Jenkins	18 November 2016	Fall	$65.34M	65.34	Yes
Parasite	Bong Joon Ho	8 November 2019	Fall	$262.69M	262.69	Yes
Say Anything	Cameron Crowe	14 April 1989	Spring	$21.52M	21.52	No
Selma	Ava DuVernay	9 January 2015	Winter	$66.79M	66.79	No
We Bought a Zoo	Cameron Crowe	23 December 2011	Winter	$120.08M	120.08	No

Case Study: Movie Wrangling

another challenge was asking something about computation (which is an important topic from the review of syllabi) but in a language agnostic way. we wrote some questions around the processes of reading in data or noticing something was wrong in the data with mock computer software displays, but all of those questions ended up being too language-specific and were removed after faculty interviews. Thus, we consolidated some of the data manipulation questions to include more explicit pseudocode: “a general recipe” of steps to result in a desired outcome that does not align exactly with any real programming language. However, this is an opportunity to instructors using this assessment to possibly tailor it to their own courses, by replacing with an actual language and testing syntax, etc.

The table below provides data about 10 movie directors. It provides data on the director’s name, the number of Oscars the movie’s director has been nominated for, and the number of Oscars the director has won.

Directors Table
director	nominations	oscars
Ava DuVernay	1	0
Barry Jenkins	3	1
Bong Joon Ho	3	3
Cameron Crowe	3	1
Enrico Casarosa	2	0
Loveleen Tandan	0	0
Nora Ephron	3	0
Penny Marshall	0	0
Sian Heder	1	1
Steven Spielberg	19	3

Case Study: Movie Wrangling

another challenge was asking something about computation (which is an important topic from the review of syllabi) but in a language agnostic way. we wrote some questions around the processes of reading in data or noticing something was wrong in the data with mock computer software displays, but all of those questions ended up being too language-specific and were removed after faculty interviews. Thus, we consolidated some of the data manipulation questions to include more explicit psuedocode: “a general recipe” of steps to result in a desired outcome that does not align exactly with any real programming language. However, this is an opportunity to instructors using this assessment to possibly tailor it to their own courses, by replacing with an actual language and testing syntax, etc.

start_with(the Movies table) then
  keep_rows_where(the season value is "Fall") then
  count(the number of rows)

start_with(the Movies table) then
  keep_rows_where(the season value is "Fall") then
  add_columns_from(the Director table) matching_by(the director column) then
  count(the number of rows) where (oscars value is 3) and (best_picture value is "No")

Assessment Next Steps

199 Pilot
IRB Roadblocks
NSF Grant

Working on the `dsbox` package

`dsbox` package

Growing interest in DS requires scalability
Data Science in a Box project
Turning it into dsbox

How does it work?

2 key packages: learnr and gradethis.
learnr: robust, broad framework.
gradethis: sophisticated testing logic.

Creating a Tutorial

9 existing, 1 started
Modifying for interactive tutorial
- Scaffolding, clear section breaks, engaging flow

Sample Tutorial: Home Page

Sample Tutorial: Code chunk with hint

Sample Tutorial: Opening the hood

```{r common-themes, exercise = TRUE}
lego_sales |>
  ___(___)
```


```{r common-themes-hint-1}
Look at the previous question for help!
```


```{r common-themes-solution}
lego_sales |>
  count(theme, sort = TRUE)
```

Sample Tutorial: Opening the hood

```{r common-themes-check}
grade_this({
  if(identical(as.character(.result[1,1]), "Star Wars")) {
    pass("You have counted themes and sorted the counts correctly.")
  }
  if(identical(as.character(.result[1,1]), "Advanced Models ")) {
    fail("Did you forget to sort the counts in descending order?")
  }
  if(identical(as.character(.result[1,1]), "Classic")) {
    fail("Did you accidentally sort the counts in ascending order?")
  }

  if(identical(as.character(.result[1,1]), "Adventure Camp")) {
    fail("Did you count subthemes instead of themes?")
  }
  if(identical(as.numeric(.result[1,2]), 172)) {
    fail("Did you count subthemes instead of themes?")
  }
  fail("Not quite. Take a peek at the hint!")
})
```

Releasing to CRAN

Comprehensive R Archive Network
Package DESCRIPTION file
gradethis still in development

Discussion

Learning Takeaways

Advanced computing
Interacting with others’ code

Reflections

“Teaching material is only way to master it”
New appreciation for existing educational materials and research
Inspired me to continue interacting with the world of open source software

Q&A

Browse at your own pace at https://evandragich.github.io/thesis-work/
Email me at evandragich@gmail.com

Exploring Data Science Education: From Tutorials to Assessment

About Me

Thesis TOC/Agenda

Building a Data Science Assessment

Background

Initial cleaning

Initial cleaning

Interviews

Interviews: Faculty

Interviews: Students

Current Prototype

Case Study: Application Screening

Case Study: Application Screening

Case Study: Data Confidentiality

Case Study: Data Confidentiality

Case Study: Data Confidentiality

Case Study: Movie Budgets 1

Case Study: Movie Budgets 1

Case Study: Movie Wrangling

Case Study: Movie Wrangling

Case Study: Movie Wrangling

Case Study: Movie Wrangling

Assessment Next Steps

Working on the dsbox package

dsbox package

How does it work?

Creating a Tutorial

Sample Tutorial: Home Page

Sample Tutorial: Code chunk with hint

Sample Tutorial: Opening the hood

Sample Tutorial: Opening the hood

Releasing to CRAN

Discussion

Learning Takeaways

Reflections

Q&A

Working on the `dsbox` package

`dsbox` package