# SIGCSE 2020 Online - Papers

## Sessions

### Session 1A: Security A

#### What Are Cybersecurity Education Papers About? A Systematic Literature Review of SIGCSE and ITiCSE Conferences

Authors: Valdemar Švábenský; Jan Vykopal; Pavel ?eleda

Abstract:

3rd Best Paper, CS Education Research Track

Cybersecurity is now more important than ever, and so is education in this field. However, the cybersecurity domain encompasses an extensive set of concepts, which can be taught in different ways and contexts. To understand the state of the art of cybersecurity education and related research, we examine papers from the ACM SIGCSE and ACM ITiCSE conferences. From 2010 to 2019, a total of 1,748 papers were published at these conferences, and 71 of them focus on cybersecurity education. The papers discuss courses, tools, exercises, and teaching approaches. For each paper, we map the covered topics, teaching context, evaluation methods, impact, and the community of authors. We discovered that the technical topic areas are evenly covered (the most prominent being secure programming, network security, and offensive security), and human aspects, such as privacy and social engineering, are present as well. The interventions described in SIGCSE and ITiCSE papers predominantly focus on tertiary education in the USA. The subsequent evaluation mostly consists of collecting students' subjective perceptions via questionnaires. However, less than a third of the papers provide supplementary materials for other educators, and none of the authors published their dataset. Our results provide orientation in the area, a synthesis of trends, and implications for further research. Therefore, they are relevant for instructors, researchers, and anyone new in the field of cybersecurity education. The information we collected and synthesized from individual papers are organized in a publicly available dataset.

#### Integration and Evaluation of Spiral Theory based Cybersecurity Modules into core Computer Science and Engineering Courses

Authors: Debarati Basu; Harinni K. Kumar; Vinod K. Lohani; N. Dwight Barnette; Godmar Back; Dave McPherson; Calvin J. Ribbens; Paul Plassmann

Abstract:

Cybersecurity education has been emphasized by several national organizations in the United States, including the National Academy of Engineering, which recognizes securing cyberspace as one of the 14 Engineering Grand Challenges. To prepare students for such challenges and to enhance cybersecurity education opportunities at our large research university, we implemented an NSF-funded cybersecurity education project. This project is a collaborative effort between faculty and graduate students in the Engineering Education, Computer Science (CS) and Computer Engineering (CPE) departments at a major US research university. In this effort, we integrated cybersecurity learning modules into multiple existing core CS and CPE courses following Jerome Bruner's spiral-theory model, which has previously been used to reformulate several academic curricula.

In this paper, we present our cybersecurity curriculum initiative, describe the spiral-theory based process we developed to implement the curriculum and provide an in-depth description of four reusable cybersecurity learning modules that we developed. A core tenet of spiral theory holds to revisit topics as students advance through their curriculum. This work applies this approach to Cybersecurity education by carefully designing the learning objectives of the modules and its contents. For evaluating these learning modules we implemented pre and post-tests to assess students' technical knowledge, their perceptions towards the modules' learning objectives, and how it influenced their motivation to learn cybersecurity. Our findings are overwhelmingly positive and the students' feedback has helped us improve these learning modules. Since its inception, our initiative has educated more than 2,000 students and is currently being used to revise the affected courses' syllabi.

#### A Case Study of a Cybersecurity Programme: Curriculum Design, Resource Management, and Reflections

Authors: Muhammad Rizwan Asghar; Andrew Luxton-Reilly

Abstract:

Cybersecurity is an area of growing international importance. In response to global shortages of Cybersecurity skills, many universities have introduced degree programmes in Cybersecurity. These programmes aim to prepare students to become Cybersecurity practitioners with advanced skills in a timely manner. Several universities offer Cybersecurity degrees, but these have been developed ad hoc, as there is currently no internationally accepted Cybersecurity curriculum.

Recently, an ITiCSE working group on global perspectives on Cybersecurity education developed a competency-based framework that aims to help institutions to implement Cybersecurity programmes. In this report, we present a case study of a Cybersecurity programme at the University of Auckland. We discuss how the curriculum and resource management of this programme evolved, and we present some challenges for the design and delivery of a Cybersecurity programme in the light of this competency-based framework.

### Session 1B: Student Experiences

#### A Longitudinal View of Gender Balance in a Large Computer Science Program

Authors: Amy Baer; Andrew DeOrio

Abstract:

Computer Science has a persistent lack of women's participation. In order to best effect change, we require a more fine-grain analysis of the gender disparity as it changes throughout an undergraduate Computer Science curriculum. In this paper, we use a quantitative approach to highlight, with greater specificity, the point in an undergraduate career where gender balance changes. We also examine the role of grades in students' decisions to stay in the course sequence. Our goal is to enable targeted interventions that will make Computer Science a more welcoming discipline. Our study examines 30,890 unique student records over ten years at a large, public research institution. The records include students who took a Computer Science course over the past ten years. The dataset contains information about gender, majors, minors, academic level, and GPA. The dataset also includes a record from each course taken by each student and their final grade. We observed a modest increase in women's participation in all Computer Science courses over the past ten years. Despite this increase, the gender disparity is still large. Through our analysis, we found that women consistently choose not to continue through the Computer Science sequence at a higher rate than men. This higher attrition could be linked to women receiving lower grades in most introductory CS courses despite having the same or higher GPAs than men. Our results reveal specific areas where intervention can be the most effective in changing the stubborn gender disparity in Computer Science.

#### Identifying the Prevalence of the Impostor Phenomenon Among Computer Science Students

Authors: Adam Rosenstein; Aishma Raghu; Leo Porter

Abstract:

The Impostor Phenomenon (IP) is often discussed as a problem in the field of computer science, but there has yet to be an empirical study to establish its prevalence among CS students. One survey by the Blind app found that a high number of software engineers at some of the largest technology companies self-reported feelings of Impostor Syndrome; however, self-reporting of Impostor Syndrome is not the standard diagnostic for identifying whether an individual exhibits feelings of the Impostor Phenomenon. In this work, the established Clance IP Scale is used to identify the prevalence of IP among graduate and undergraduate computer science students at a large research-intensive North American institution. Among this population of over 200 students, 57% were found to exhibit frequent feelings of the Impostor Phenomenon with a larger fraction of women (71%) experiencing frequent feelings of the Imposter Phenomenon than men (52%). Additionally, IP was found to have greater prevalence among computer science students than among students of other populations from comparable studies. Due to the negative impacts associated with feelings of the Impostor Phenomenon, computer science education should work to improve student awareness and help student cope with these feelings.

#### A Multi-level Analysis of the Relationship between Instructional Practices and Retention in Computer Science

Authors: Markeya Peteranetz; Leen-Kiat Soh

Abstract:

Increasing retention in computer science (CS) courses is a goal of many CS departments. A key step to increasing retention is to understand the factors that impact the likelihood students will continue to enroll in CS courses. Prior research on retention in CS has mostly examined factors such as prior exposure to programming and students' personality characteristics, which are outside the control of undergraduate instructors. This study focuses on factors within the control of instructors, namely, instructional practices that directly impact students' classroom experiences. Participants were recruited from 25 sections of 14 different courses over 4 semesters. A multi-level model tested the effects of individual and class-average perceptions of cooperative learning and teacher directedness on the probability of subsequent enrollment in a CS course, while controlling for students' mastery of CS concepts and status as a CS major. Results indicated that students' individual perceptions of instructional practices were not associated with retention, but the average rating of cooperative learning within a course section was negatively associated with retention. Consistent with prior research, greater mastery of CS concepts and considering or having declared a CS major were associated with a higher probability of taking a future CS courses. Implications for findings are discussed.

### Session 1C: Automated Systems

#### Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style

Authors: Abe Leite; Saúl A. Blanco

Abstract:

The use of automatic grading tools has become nearly ubiquitous in large undergraduate programming courses, and recent work has focused on improving the quality of automatically generated feedback. However, there is a relative lack of data directly comparing student outcomes when receiving computer-generated feedback and human-written feedback. This paper addresses this gap by splitting one 90-student class into two feedback groups and analyzing differences in the two cohorts' performance. The class is an intro to AI with programming HW assignments. One group of students received detailed computer-generated feedback on their programming assignments describing which parts of the algorithms' logic was missing; the other group additionally received human-written feedback describing how their programs' syntax relates to issues with their logic, and qualitative (style) recommendations for improving their code. Results on quizzes and exam questions suggest that human feedback helps students obtain a better conceptual understanding, but analyses found no difference between the groups' ability to collaborate on the final project. The course grade distribution revealed that students who received human-written feedback performed better overall; this effect was the most pronounced in the middle two quartiles of each group. These results suggest that feedback about the syntax-logic relation may be a primary mechanism by which human feedback improves student outcomes.

#### Automated Assessment of the Visual Design of Android Apps Developed with App Inventor

Authors: Igor Solecki; João Porto; Nathalia Alves; Christiane Gresse von Wangenheim; Jean Hauck; Adriano Ferreti Borgatto

Abstract:

One way to teach computing in K-12 is through the development of mobile applications with App Inventor. Although already broadly used worldwide, there is still a need for support for the assessment of the applications created by the students. Existing rubrics focusing mostly on programming concepts do not cover more comprehensively the performance-based assessment of user interface design concepts, important for the usability and aesthetics of the applications. Thus, in order to support the assessment of the visual design of apps based on its compliance with design theory and guidelines, we developed the CodeMaster UI Design - App Inventor rubric in the context of computing education. In order to facilitate its application in practice, we automated the assessment of applications created with App Inventor through a static code analysis by an online tool. We evaluated the reliability and validity of the rubric based on the automated assessment of 1,775 projects from the App Inventor Gallery. The results indicate that the rubric can be considered reliable (Cronbach's alpha = 0.84). In terms of construct validity, there is also evidence of convergent validity. The results presented in this article can be used to support the assessment of computing education in practice as well as to point out further research opportunities.

#### Development and Adoption of an Adaptive Learning System: Reflections and Lessons Learned

Authors: Hassan Khosravi; Shazia Sadiq; Dragan Gasevic

Abstract:

Adaptive learning systems (ALSs) aim to provide an efficient, effective and customised learning experience for students by dynamically adapting learning content to suit their individual abilities or preferences. Despite consistent evidence of their effectiveness and success in improving student learning over the past three decades, the actual impact and adoption of ALSs in education remain restricted to mostly research projects. In this paper, we provide a brief overview of reflections and lessons learned from developing and piloting an ALS in a course on relational databases. While our focus has been on adaptive learning, many of the presented lessons are also applicable to the development and adoption of educational tools and technologies in general. Our aim is to provide insight for other instructors, educational researchers and developers that are interested in adopting ALSs or are involved in the implementation of educational tools and technologies.

### Session 1D: Learning A

#### Lessons Learned From Introducing Preteens in Parent-Led Homeschooling to Computational Thinking

Authors: Carla Sepúlveda-Díaz; Elson Stuardo Rojas; Jocelyn Simmonds; Francisco J. Gutierrez; Nancy Hitschfeld; Cecilia Casanova; Cecilia Sotomayor

Abstract:

Parents that homeschool their children ignore certain topics when they lack mastery or interest in them. Homeschool groups try to address this issue, cooperatively educating their children. We were contacted by such a group that wanted to introduce their children to computational thinking (CT). These children, aged 7-11, have showed an interest in technology, and use online educational resources. None of the parents felt capable of tutoring the group about CT. They also worried about losing control about how their children interact with technology. We report an intervention over 9 months to introduce eleven young homeschoolers to CT in a suburban environment, describing the impact on parent and children attitudes towards technology and CT. We conclude with three lessons: 1) science-related activities should be used to introduce CT among homeschoolers, 2) "success" is establishing a meaningful relationship with a homeschool group, and 3) activities designed for school children need to be adapted to the homeschooling context.

#### Understanding Wikipedia as a Resource for Opportunistic Learning of Computing Concepts

Authors: Martin P. Robillard; Christoph Treude

Abstract:

Posts on on-line forums where programmers look for information often include links to Wikipedia when it can be assumed the reader will not be familiar with the linked terms. A Wikipedia article will thus often be the first exposure to a new computing concept for a novice programmer. We conducted an exploratory study with 18 novice programmers by asking them to read a Wikipedia article on a common computing concept that was new to them, while using the think-aloud protocol. We performed a qualitative analysis of the session transcripts to better understand the experience of the novice programmer learning a new computing concept using Wikipedia. We elicited five themes that capture this experience: Concept Confusion, Need for Examples, New Terminology, Trivia Clutter, and Unfamiliar Notation. We conclude that Wikipedia is not well suited as a resource for the opportunistic learning of new computing concepts, and we recommend adapting information sharing practices in on-line programmer communities to better account for the learning needs of the users.

#### TIPP&SEE: A Learning Strategy to Guide Students through Use ? Modify Scratch Activities

Authors: Jean Salac; Cathy Thomas; Chloe Butler; Ashley Sanchez; Diana Franklin

Abstract:

With the rise of Computational Thinking (CT) instruction at the elementary level, it is imperative that elementary computing instruction support a variety of learners. A popular pedagogical approach for this age group is Use-->Modify-->Create, which introduces a concept through a more scaffolded, guided instruction before culminating in a more open-ended project for student engagement. Yet, there is little research on student learning during the Use-->Modify step, nor strategies to promote learning in this step. This paper introduces TIPP&SEE, a metacognitive learning strategy that further scaffolds student learning during this step. Results from an experimental study show statistically-significant performance gains from students using the TIPP&SEE strategy on nearly all assessment questions of moderate and hard difficulty, suggesting its potential as an effective CS/CT learning strategy.

### Session 1E: CS 2

#### MAADS: Mixed-Methods Approach for the Analysis of Debugging Sequences of Beginner Programmers

Authors: Chaima Jemmali; Erica Kleinman; Sara Bunian; Mia Victoria Almeda; Elizabeth Rowe; Magy Seif El-Nasr

Abstract:

be especially problematic for beginners. While there has been some work trying to understand the difficulties that beginners face with debugging, investigating common mistakes or specific error types they struggle with, there is little work that focuses on in-depth analysis of how novice programmers approach debugging, and how it changes over time. In this paper, we present MAADS (Mixed-Methods Approach for the Analysis of Debugging Sequences), a scalable and generalizable approach that combines quantitative and qualitative methods by using a state/action representation and visualization to gain knowledge about the debugging process through a step by step analysis. To demonstrate the utility of MAADS, we analyzed the debugging processes of middle school students who developed code within May's Journey, a game designed to teach basic programming principles. The approach showed great utility in identifying differences in students' debugging techniques and learning paths.

#### Analyzing the Effects of Active Learning Classrooms in CS2

Authors: Ayesha Syeda; Rutwa Engineer; Bogdan Simion

Abstract:

Active learning environments have only recently started to be analyzed in the CS discipline, in terms of their effect on student performance. Recent studies in CS1 found contradictory results, in part due to different control on the learning pedagogy used, and issued a call for further investigation. This study evaluates the effects of the learning space on student performance in CS2, as measured by their grades. We use a quasi-experimental setup with 529 participants across five lecture sections over one academic term. All sections employ the same active learning method (inverted classroom), identical lecture materials, and the same number of TAs for in-class support, but differ in terms of classroom type (active learning classroom vs traditional lecture hall), instructor, and lecture time of day. Similarly to a recent study in CS1, we find no significant impact of the learning space in CS2. We also inspect factors not analyzed in previous studies, such as student prior preparation (as measured by prerequisite CS1 grades), course drop rates, and exam failure rates, and find that the CS2 sections are statistically similar.

This work also examines student survey responses, to assess student perception differences on properties of the learning space which may impact their learning experience, such as the use of technology, ability to hear the instructor, ability to get help during lectures, and conduciveness of desk types to group work.

#### Does Story Programming Prepare for Coding?

Authors: Jennifer Parham-Mocello; Martin Erwig

Abstract:

In this research study, we investigate the impact of using the Story Programming approach to teach CS concepts on student performance in a subsequent C++ class. In particular, we compare how students receiving little or no coding to learn and apply these concepts perform in comparison to students who learn these concepts only in the context of coding. While past research has shown that exposure to programming is not a predictor of success in such courses, these studies are based on a 15-week versus 10-week course and do not control for the CS concepts and programming to which the students have been exposed. Consequently, we hypothesize that students from the Story Programming approach will perform worse in the following C++ class. Surprisingly, we find that this is not true: Students from the Story Programming approach with little to no coding do not significantly differ from their peers receiving a traditional code-focused approach.

### Session 1F: Accessibility

#### A Systematic Analysis of Accessibility in Computing Education Research

Authors: Catherine M. Baker; Yasmine El-Glaly; Kristen Shinohara

Abstract:

Recent interest in accessibility emphasizes including it in computer science curriculum as key to producing effective computing professionals. Despite a general consensus that teaching accessibility in computing curriculum is good, there exist few tools and resources to support instructors in higher education. To better understand the relationship between accessibility in curriculum and research, we conducted a systematic literature review of papers in computing education. We analyzed the papers for the courses accessibility is covered in, the topics that are covered and pedagogies and assessment approaches that are used. Across this body of work, we found a number of key learning objectives commonly covered in computing education research, though it appeared the research did not evenly cover these objectives throughout curricula, nor did the research systematically investigate how learning objectives were integrated. Based on these results, we offer suggestions for future directions of accessibility education research and curriculum building.

#### Participation at What Cost? Teaching Accessibility Using Participatory Design: An Experience Report

Authors: Julian Brinkley

Abstract:

As institutions respond to market demand in their training of the next generation of technology designers, there is an increasing awareness of the need to add accessibility to computer science and informatics curricula. Advocates have suggested three strategies for including accessibility and discussions of disability in courses: changing a lecture, adding a lecture or adding a new course. In this paper we report on our experiences with the latter; incorporating accessibility within two new graduate and undergraduate inclusive design courses taught concurrently. We found that while the use of participatory design was decidedly effective in supporting student learning and ameliorating ableist attitudes, creating and managing teams comprised of students and visually impaired co-designers proved challenging. Despite these challenges, overall, students demonstrated steady growth in their grasp of inclusive design concepts as they tackled accessibility challenges through a series of mobility-related group projects. Efficiencies were also realized through the concurrent teaching of both courses though the pace of course deliverables proved challenging at times for undergraduates. We argue that a review of our experience may help others interested in teaching accessibility related courses, specifically in course design and execution.

#### Teaching Accessibility to Software Engineering Students

Authors: Yasmine El-Glaly; Catherine M. Baker

Abstract:

This paper describes the development process of a graduate course on accessibility that is offered as an elective for software engineering students. The paper reports on the three iterations of the course evolution as topics and pedagogy are updated from one iteration to the next. The main motivation of the course updates was to cover the educational needs of software engineering students, which are not the same as HCI or design students. Software engineering students learned better and became more engaged with the topic of accessibility when it was tied to programming and development- based activities. The final form of the course was evaluated using a survey, and the results showed that students found the course beneficial to their education and relevant to their career. We discuss the challenges of creating, teaching, and maintaining a course on accessibility, and we offer insights on what research is needed in this area to support accessibility educators.

### Session 1G: Mentoring

#### The Impact of Undergraduate Mentorship on Student Satisfaction and Engagement, Teamwork Performance, and Team Dysfunction in a Software Engineering Group Project

Authors: Claudia Iacob; Shamal Faily

Abstract:

Mentorship schemes in software engineering education usually involve professional software engineers guiding and advising teams of undergraduate students working collaboratively to develop a software system. With or without mentorship, teams run the risk of experiencing team dysfunction: a situation where lack of engagement, internal conflicts, and/or poor team management lead to different assessment outcomes for individual team members and overall frustration and dissatisfaction within the team. The paper describes a mentorship scheme devised as part of a 33 week software engineering group project course, where the mentors were undergraduate students who had recently completed the course successfully and possessed at least a year's experience as professional software engineers. We measure and discuss the impact the scheme had on: (1) student satisfaction and engagement, (2) team performance, and (3) team dysfunction.

#### Formal Methods and Computing Identity-based Mentorship for Early Stage Researchers

Authors: Mark Santolucito; Ruzica Piskac

Abstract:

The field of formal methods relies on a large body of background knowledge that can dissuade researchers from engaging with younger students, such as undergraduates or high school students. However, we have found that formal methods can be an excellent entry point to computer science research - especially in the framing of Computing Identity-based Mentorship. We report on our experience in using a cascading mentorship model to involve early stage researchers in formal methods, covering our process with these students from recruitment to publication. We present case studies (N=12) of our cascading mentorship and how we were able to integrate formal methods research with the students' own interests. We outline some key strategies that have led to success and reflect on strategies that have been, in our experience, inefficient.

#### Undergraduate-Led Survey Class to Improve CS Education for New Students

Authors: Nathan Zhang; Jacky Liang; Amanda Tomlinson; Frank Boensch; Anant Sahai

Abstract:

Many first-year undergraduate students do not have sufficient breadth of technical knowledge about subjects in Electrical Engineering (EE) and Computer Science (CS) to make informed choices toward their education. By the time students are exposed to subjects they may be interested in, the cost of switching areas of focus may be too high. With undergraduate enrollment in CS more than doubling in the past decade, many institutions lack adequate staff and infrastructure to address students' needs. To help newly enrolled students make better decisions with limited departmental resources, we present a first-semester, low-overhead survey course that covers a wide variety of topics. The class has been offered for five consecutive semesters by upper-class undergraduate volunteers with minimal faculty involvement. We report the format, content, and student feedback for the course. Our results suggest that such a class can provide new students with valuable guidance and better prepare them for an education in CS and EE.

### Session 1H: Problem Solving

#### Investigating Novices' In Situ Reflections on Their Programming Process

Authors: Dastyni Loksa; Benjamin Xie; Harrison Kwik; Amy J. Ko

Abstract:

Prior work on novice programmers' self-regulation have shown it to be inconsistent and shallow, but trainable through direct instruction. However, prior work has primarily studied self-regulation retrospectively, which relies on students to remember how they regulated their process, or in laboratory settings, limiting the ecological validity of findings. To address these limitations, we investigated 31 novice programmers' self-regulation in situ over 10 weeks. We had them to keep journals about their work and later had them to reflect on their journaling. Through a series of qualitative analyses of journals and survey responses, we found that all participants monitored their process and evaluated their work, that few interpreted the problems they were solving or adapted prior solutions. We also found that some students self-regulated their programming in many ways, while others in almost none. Students reported many difficulties integrating reflection into their work; some were completely unaware of their process, some struggled to integrate reflection into their process, and others found reflection conflicted with their work. These results suggest that self-regulation during programming is highly variable in practice, and that teaching self-regulation skills to improve programming outcomes may require differentiated instruction based on students self-awareness and existing programming practices.

#### Can Majoring in Computer Science Improve General Problem-solving Skills?

Authors: Shima Salehi; Karen D. Wang; Ruqayya Toorawa; Carl Wieman

Abstract:

Teaching students to become skillful problem solvers is a goal of university education, but it has been difficult to measure such skill or demonstrate the benefits of particular educational experiences. This paper presents a study of college students solving a problem unrelated to their academic majors. The analysis suggests that the educational experiences of Computer Science (CS) students may better train them in problem-solving than the experiences of other majors. In this study, students from a variety of undergraduate majors and grade levels were given a 15-minute problem-solving task embedded in an interactive science simulation. The complex task calls upon many problem-solving practices needed by scientists and engineers in their professions. Although this task has little resemblance to the problems encountered in a computer science course, CS students performed significantly better than students in any other major. In addition, only for CS students was there an indication of improvement in problem-solving from lower to upper grade levels. We propose that general problem-solving and computational thinking share some common practices, such as problem decomposition and comprehensive data collection. Furthermore, we present preliminary evidence that training in computational thinking is transferable to problem-solving tasks across domains and discuss how the unique features of CS programming assignments could be generalized to other science and engineering courses to foster students' general problem-solving skills.

#### Investigating Patterns of Study Persistence on Self-Assessment Platform of Programming Problem-Solving

Authors: Cheng-Yu Chung; I-Han Hsiao

Abstract:

A student's short-term study behavior may not necessary infer his/her long-term behavior. It is very common to see a student changes study strategy throughout a semester and adapts to learning condition. For example, a student may work very hard before the first exam but gradually reducing the effort due to several possible reasons, e.g., being overwhelmed by various course work or discouraged by increasing complexity in the subject. Consistency or differences of one student's behavior is more likely to be discovered by multiple granularity of learning analytics. In this study, we investigate students' study persistence on a self-assessment platform and explore how such a behavioral pattern is related to the performance in exams. A probabilistic mixture model trained by response streams of log data is applied to cluster students' behavior into persistence patterns, which are further categorized into "micro" (short-term) and "macro" (long-term) patterns according to the span of time being modeled. We found four types of micro persistence patterns and several macro patterns in the analysis and analyzed their relations with exam performances. The result suggests that the consistency of persistence patterns can be an important factor driving student's overall performance in the semester, and students achieving higher exam scores show relatively persistent behavior compared to students receiving lower scores.

### Session 2A: Data Science A

#### A Data Science Major: Building Skills and Confidence

Authors: Stephanie Rosenthal; Tingting Chung

Abstract:

Data science is a growing field at the intersection of mathematics, computer science, and domain expertise. Like many universities that are building data science degree programs for undergraduates, our small, liberal-arts university saw increasing opportunities in the region and decided to build a data science degree from the ground up, without a pre-existing computer science (CS) department to leverage for courses or culture. We designed and implemented an academically-demanding curriculum that combined mathematics, information systems, and new data science courses, and that also encouraged and supported student success. Each introductory course included active learning design to engage students. To increase retention, all major courses included assignments designed to build skills but also student confidence in their ability to learn challenging technical topics. Outside of the classroom, we created opportunities for professional advancement and developed a technical culture at the university. We will share our approach, course highlights, and lessons learned from building such a curriculum at an institution without a CS department.

#### Creating a Balanced Data Science Program

Authors: Joel Adams

Abstract:

3rd Best Paper, Curricula Initiatives

As we consider the next fifty years of computing education, a phenomenon that shows no signs of abating is the data deluge, in which commercial companies, the natural sciences, the social sciences, professional sports teams, government agencies, and other institutions are generating ever-increasing quantities of data. To address the challenges posed by the data deluge, the discipline of data science has arisen, and an increasing number of universities are offering undergraduate data science programs. Many of these programs have their origins in a computer science or a statistics department, leading to a data science curriculum that is more heavily weighted toward computing or statistics. By contrast, the data science program described in this paper is a joint endeavor between computer science and statistics that seeks to provide balanced training in both areas. Its broad goals are to produce students who (a) are well-trained in both computer science and statistics, (b) are equipped with specialized data-related skills that are not normally taught in either of those disciplines, and (c) can apply their skills to a domain area. This paper reports on the author's experiences leading the effort to create this program, which has seen good growth, received positive feedback from students, and is successfully preparing students for internships. We offer this report in the hope that it may serve as a model for other institutions considering the addition of an undergraduate data science program.

#### Insights for Curriculum Development: Identifying Emerging Data Science Topics through Analysis of Q&A Communities

Authors: Habib Karbasian; Aditya Johri

Abstract:

Updating curricula in new computer science domains is a critical challenge faced by many instructors and programs. In this paper we present an approach for identifying emerging topics and issues in Data Science by using Question and Answer (Q&A) sites as a resource. Q&A sites provide a useful online platform for discussion of topics and through the sharing of information they become a valuable corpus of knowledge. We applied latent Dirichlet allocation (LDA), a statistical topic modeling technique, to analyze data science related threads from from two popular Q&A communities "Stack Exchange and Reddit". We uncovered both important topics as well as useful examples that can be incorporated into teaching. In addition to technical topics, our analysis also identified topics related to professional development. We believe that approaches such as these are critical in order to update curriculum and bridge the workplace-school divide in teaching of newer topics such as data science. Given the pace of technical development and frequent changes in the field, this is an inventive and effective method to keep teaching up to date. We also discuss the limitations of this approach whereby topics of importance such as data ethics are largely missing from online discussions.

### Session 2B: Diversity

#### The Intersectional Experiences of Black Women in Computing

Authors: Yolanda Rankin; Jakita O. Thomas

Abstract:

Efforts to broaden participation in computing have led to gender-focused interventions intended to increase the number of women in the field of computing. However, such efforts have failed to significantly increase the percentage of Black women in computing. For example, only 1% of the 28,884 bachelor's degrees in computing were awarded to Black women in 2018. Moreover, too few empirical studies have intentionally explored the lived experiences of Black women, an often overlooked and understudied population in the computing ecosystem. In this paper, we introduce intersectionality - the complex overlap of socially constructed identities such as race, gender, class, sexuality, etc. - as a theoretical framework and springboard for exploring the lived experiences of Black women in computing. We interview 14 Black women in various stages of the computing ecosystem (undergraduate students, graduate students and early career professionals) to understand how intersectionality influences their ability to persist in computing. Preliminary findings from the analysis of the 14 interviews provides insights into how the interlocking systems of oppression (i.e., gendered racism) play out in computing education and negatively impact the recruitment and retention of Black women in the field of computing.

#### Lessons Learned from Providing Hundreds of Hours of Diversity Training

Authors: Luther Tychonievich; James P. Cohoon

Abstract:

The past thirty years have seen many advances in our understanding of issues that impede diversity, as well as interventions that can remove or mitigate those impediments. The advances are generally presented in psychology journals and are under-consumed by computing educators, leading to the need for diversity training tailored to said educators. We have hundreds of hours of experience providing diversity training to computing educators, and have learned many lessons about how training should and should not be delivered to engage participants in understanding diversity-impeding forces and implementing interventions to improve the attraction and retention of diverse students.

This paper is a collection of those lessons learned. We focus on concrete approaches to presenting material rather than on training organization or content, including a dozen specific practices that have proven particularly effective or ineffective in reaching diverse audiences. Examples include how how to avoid offending participants, how to help experts not lose their audience, how to pace and organize material for maximum impact, how we tie many topics into a single cognitive framework, and how we've learned to handle participant-raised objections. We also include commentary as to why we believe they did or did not work. We also describe possible future research to validate and expand on our observations and a few open questions for diversity training.

#### When Twice as Good Isn't Enough: The Case for Cultural Competence in Computing

Authors: Nicki Washington

Abstract:

The commonly documented diversity, equity, and inclusion (DEI) issues in the computing workforce are the direct result of corporate cultures that benefit specific groups and marginalize others. This culture usually begins in undergraduate computing departments, where the demographic representation mirrors that of industry. With no formal courses that focus on the non-technical issues affecting marginalized groups and how to address and eradicate them, students are indirectly taught that the current status quo in computing departments and industry is not only acceptable, but also unproblematic. This directly affects students from marginalized groups (as the reasons for attrition are similar in both higher education and industry), as well as faculty (as biased student evaluations directly affect hiring, promotion, and tenure decisions).

This position paper presents the need for cultural competence as a required focus for university computing departments nationwide. By improving these issues before students complete baccalaureate computing degrees, companies will have talent pools that better understand the importance and necessity of DEI and also work to ensure they help foster a more diverse, equitable, and inclusive environment. In addition, more students from marginalized groups will be retained in the major through degree completion.

### Session 2C: CS0

#### Design Principles behind Beauty and Joy of Computing

Authors: Paul Goldenberg; June Mark; Brian Harvey; Al Cuoco; Mary Fries

Abstract:

Best Paper, Curricula Initiatives

This paper shares the design principles of one Advanced Placement Computer Science Principles (AP CSP) course, Beauty and Joy of Computing (BJC), both for schools considering curriculum, and for developers in this still-new field. BJC students not only learn about CS, but do some and analyze its social implications; we feel that the job of enticing students into the field isn't complete until students find programming, itself, something they enjoy and know they can do, and its key ideas accessible. Students must feel invited to use their own creativity and logic, and enjoy the power of their logic and the beauty and elegance of the code by which they express it. All kids need genuine challenge and sensible support so all can have the joy of making-seeing themselves as creators, not just consumers, and seeing that it is their own intellect, not just our instructions, that is the source of that making. Framework standards are woven into a consistent social and intellectual storyline to give the curriculum integrity.

Principles guide even our choice of programming language. Learners should focus on the logic and structure of their thinking, not on misplaced semicolons; attention to such syntactic detail is antithetical to broadening participation. We feature recursion and higher-order functions because they beautifully exemplify abstraction, a key idea in CS and the CSP framework.

BJC also places significant emphasis on the social implications of computing, balancing fundamental optimism about computing technology with a critical view of specific uses of technology.

#### CS 0: Culture and Coding

Authors: Albert Lionelle; Josette Grinslad; J. Ross Beveridge

Abstract:

In 2018, Colorado State University redesigned their CS-0 course to become a general education requirement for the university within Arts and Humanities, and a guaranteed transfer course across the state for a similar category in other universities. The first CS course in the State to be accepted as a GT-Pathway course. This redesign had to be carefully done due to a need to introduce liberal arts style topics such as CS History, Philosophy and Ethics and Inclusive Design issues, while maintaining current coding and student success standards that were already expected for the CS-0 at the university. We termed this combination as Culture and Coding. In order to add more without reducing retention, the course was redesigned around the Psychology of Learning and spacing of topics in a "Spiral" Manner. Each topic was briefly introduced, and throughout the semester, students would dive deeper into the topics. This allowed for a 50% reduction of time focused on teaching coding topics, with students performing equivalent on exams compared to previous models of the course that focused 100% of the time on coding topics. Furthermore, students taught by the spiral teaching method outperformed students taught using traditional methods in the follow-on course. Our evaluation suggests that the spiral model of teaching computer science may allow for greater retention of topics, allowing classes to either cover additional concepts or go more in depth on current topics.

#### Introducing Computing to a Cohort of Incarcerated Youth

Authors: Kirsten Mork; Theresa Migler; Zoë Wood

Abstract:

Computer Science and programming are changing the world, but not everyone has equal access to education about this field. In California, Juvenile hall students typically lack opportunities to learn computer programming. In this paper, we present an experience report about the course we created to address the needs of this population. Specifically, we created and taught an introductory computing course focused on engagement. This project-based game design curriculum was launched to a small cohort of the juvenile hall in spring 2019. This course focused on engaging students while introducing computing programming concepts such as variables, logic and function. Student surveys and reports from their teacher showed this class had a positive impact and was well received by students and staff. We hypothesize and show initial positive indication that creative, game-oriented curriculum had a positive impact on the demographic. We also present some of the challenges encountered when working within the juvenile hall system and our solutions and general recommendations for these types of classes.

### Session 2D: BPC Initiatives

#### Algebra I Before High School as a Gatekeeper to Computer Science Participation

Authors: Ryan Torbey; Nicole D. Martin; Jayce R. Warner; Carol Fletcher

Abstract:

A complex web of factors can influence whether students participate in computer science (CS) during high school. In order to increase participation in CS for all students, we need to better understand who is currently participating and what factors might be hindering participation. This study utilized a large-scale, student-level dataset from the Texas Education Research Center to investigate factors that predict high school student participation in CS and advanced CS courses. Our dataset contained information on over 1.1 million Texas high school students from the 2017-2018 school year, allowing us visibility into CS course availability in schools, student course taking, and detailed demographic information. We used multilevel mixed-effects logistic regression models to explore predictive factors of student participation in CS and advanced CS courses, limiting our analysis to students whose schools offered CS. In both models, our results showed that students who took Algebra I before high school had more than double the odds of being enrolled in a CS course. This work supports and extends previous understanding of factors that are predictive of CS participation in high school, contributing to the existing literature by uncovering the importance of Algebra I before high school as a potential gatekeeper to participation in CS.

#### Promoting and Supporting Computer Science Among Middle School Girls of Color

Authors: Veronica Madrigal; Ryoko Yamaguchi; Adam Hall; Jamika Burge

Abstract:

BRIGHT-CS (Building Student Retention through Individuated Guided coHort Training in Computer Science) is a research and development project that 1) creates a computer science learning ecosystem for middle school Black girls and other girls of color and 2) researches the merits of the ecosystem in supporting persistence in CS to determine best practices for broadening participation to other marginalized student groups in computing. First, this paper describes the BRIGHT-CS program, from the structural, instructional, and curricular designs of the program to partnerships with local and community organizations that make up the ecosystem. Second, it presents the initial findings of research on the program and its impacts on student outcomes such as social-emotional attributes associated with persistence. The study employs a multi-method descriptive design. Data includes student surveys, interviews (from students, parents, instructors, teachers, and mentors), artifact reviews, and student observations. The study includes 46 students across four middle schools in two states. At the start of the program, 37% of the students reported being very interested in CS, and 72% reported being very confident in learning CS. This is much higher than a national benchmark of students. After four months of program implementation, the qualitative results show a more nuanced picture of the value of a learning ecosystem. First, the ecosystem offers implicit messaging about equity and success. Second, the ecosystem offers explicit messaging about personal challenges and improvement. Third, following the implicit and explicit messaging to students, students went from naïve confidence to authentic self-efficacy in CS.

#### Broadening Participation in Computing: The Role of Upward Transfer

Authors: Jennifer Blaney

Abstract:

Broadening participation in computing (BPC) has received increasing attention within the computing education community. Yet, much of the existing research on this topic focuses primarily on students who follow traditional pathways to four-year computing programs. This multi-institution study addresses this gap in the literature, providing insight into how upward transfer students (i.e., students who transfer from community colleges to four-year computing programs) fit into the larger BPC narrative. Specifically, I use survey data from computing majors across 15 universities to examine how the characteristics and experiences of upward transfer students differ from those of other computing majors. Findings reveal that while upward transfer students tend to be more diverse than other computing majors in terms of race/ethnicity, socioeconomic status, and other background characteristics, women are even more underrepresented among upward transfers compared to computing majors who began their degree at four-year institutions. These and other findings are discussed, focusing on implications for how institutions can support upward transfer students as part of their larger BPC efforts.

### Session 2E: Math & CS

#### Didactic Methods of Integrating Programming in Mathematics in Primary School: Findings from a Swedish National Project

Authors: Gashawa Ahmed; Jalal Nouri; leChen Zhang; Eva Norén

Abstract:

The association between mathematics and programming in an educational context is not new. Today, programming has been introduced into curricula worldwide for younger children. In the Swedish case, primary school teachers are expected to integrate programming in mathematics education from autumn 2018. However, Swedish teachers' knowledge of programming and programming didactics is limited. Meanwhile, there is little research on K-9 programming education. This has led to the dilemma that the mathematics teachers have limited support in didactic knowledge and good examples. This study reports on a teacher professional development project in programming. More specifically, teachers used Lesson Study to plan, execute, and evaluate lessons that integrated programming into various school subjects in elementary school. This study analyzed the didactic strategies developed in 10 lesson studies, as well as mapped the opportunities and challenges of pupils' learning in the mathematics subject. The result was the identification of three didactic strategies, which were analog programming, robot programming and block programming, as well as 11 didactic methods applied within these strategies. The paper contributes with examples of the didactic methods that teachers have developed and evaluated using lesson study. The paper further provides insights on how teachers can take progression into account by applying the three didactic strategies. At last but not least, the study shows a great need for teachers to develop computational thinking abilities.

#### Comprehending Code: Understanding the Relationship between Reading and Math Proficiency, and 4th-Grade CS Learning Outcomes

Authors: Jean Salac; Cathy Thomas; Bryan “BT” Twarek; Bill Marsland; Diana Franklin

Abstract:

As many school districts nationwide continue to incorporate Computer Science (CS) and Computational Thinking (CT) instruction at the K-8 level, it is crucial that we understand the factors and skills, such as reading and math proficiency, that contribute to the success of younger learners in a computing curriculum and are typically developed at this age. Yet, little is known about the relationship between reading and math proficiency, and the learning of key CS concepts at the elementary level. This study focused on 4th-grade students (ages 9-10) who were taught events, sequence, and repetition through an adaptation of the Creative Computing Curriculum. While all students benefited from access to such a curriculum, there were statistically-significant differences in learning outcomes, especially between students whose reading and math proficiency are below grade-level, and students whose proficiency are at or above grade-level. This performance gap suggests the need for curricular improvement and learning strategies that are CS specific for students who struggle with reading and math.

#### Motivating Future Adventures in Computing by Unmasking Math Behind Movie Special Effects

Authors: Katie Chea; Christine Moore; William Bares

Abstract:

Children and adolescents consume increasing hours of screen time, the largest share of which goes to viewing movies according to Common Sense Research. Recently, movies with the highest box office revenues typically feature computer-generated imagery (CGI). Despite this substantial consumption of computer-generated imagery, the typical young viewer is unaware of how large a role mathematics plays in creating these movies. Studies have shown that there is a link between proficiency in math and predicted success in computing. In this paper, we describe our efforts to develop and evaluate CS Unplugged and code-based activities that attempt to explicitly connect math concepts with the creation of movie special effects magic. The learning activities have been designed and evaluated with middle school and first-year college students to demonstrate their potential to increase awareness and knowledge of how math is used in movie effects and to increase desire for future study of math. The paper describes the motivating context, pedagogical strategies, math learning outcomes, and CS Unplugged and code-based activities so that others may apply the methods. The paper concludes with a presentation and analysis of results from a five-day STEM summer camp for middle-school children and activities presented in first-year college classrooms and discussion of limitations and directions for future work.

### Session 2F: Ethics & Academic Integrity

#### Teaching Academic Honesty in CS50

Authors: David J. Malan; Brian Yu; Douglas Lloyd

Abstract:

We aspire to teach academic honesty in CS50 at Harvard University not only by addressing academic dishonesty when it occurs but by addressing it before it does. By way of communication, course- and campus-wide awareness of policy, just-in-time prompts, interventional conversations, and problem sets have we tried to preempt submission of plagiarized work. But few interventions have had significant or lasting effects on the number of instances thereof. Most impactful has been the addition of one sentence to the course's syllabus, a "regret clause" that encourages students to come forward within 72 hours of some dishonest act on their part, before the course itself is even aware. While we might zero the work in question in such cases, we commit to not escalating the matter further to the university's honor council, where the outcome might instead be admonishment, probation, or even required withdrawal from the university itself. We instead advise students on how best to move forward and connect them as needed with support structures on campus for academics and mental health. Since 2014 have 89 students invoked the clause, between 1% and 3% of the course's student body each year.

#### What Do We Teach When We Teach Tech Ethics? A Syllabi Analysis

Authors: Casey Fiesler; Natalie Garrett; Nathan Beard

Abstract:

As issues of technology ethics become more pervasive in the media and public discussions, there is increasing interest in what role ethics should play in computing education. Not only are there more standalone ethics classes being offered at universities, but calls for greater integration of ethics across computer science curriculum mean that a growing number of CS instructors may be including ethics as part of their courses. To both describe current trends in computing ethics coursework and to provide guidance for further ethics inclusion in computing, we present an in-depth qualitative analysis of 115 syllabi from university technology ethics courses. Our analysis contributes a snapshot of the content and goals of tech ethics classes, and recommendations for how these might be integrated across a computing curriculum.

#### Teaching Computer Ethics: A Deeply Multidisciplinary Approach

Authors: Rob Reich; Mehran Sahami; Jeremy M. Weinstein; Hilary Cohen

Abstract:

We report on a curricular experiment at Stanford University focused on teaching computer ethics. After nearly a year of preparation, we launched a new course at the intersection of ethics, public policy, and technology that deeply marries the humanities, social sciences, and computer science.

While the teaching of computer ethics courses dates back decades, such courses are often taught by a (single) CS faculty member without significant training in ethics, do not include a policy component, and are meant for CS students. By contrast, we take a deeply multidisciplinary approach, where three faculty instructors, from philosophy, political science, and CS, each bring their respective lens to four related course modules: algorithmic decision-making, data privacy and civil liberties, AI and autonomous systems, and the power of platform companies. Panels of guest speakers drawn from academia, industry, civil society, and government provide a practitioner's view of the topics addressed. Additionally, custom case studies were developed under the direction of the course staff. These materials (videos of the speaker panels and the case studies) are freely available for use by the broader community.

We report on the details of the course structure, including how multiple disciplines are integrated throughout the course, including lectures, discussions, and assignments. We discuss aspects of the course that worked well as well as challenges in making the course broadly accessible (beyond just CS majors). Importantly, we also include a discussion of students' response to the course, showing that a deeply multidisciplinary approach resonates strongly with them.

### Session 2G: Student Motivation

#### Influence of Course Design on Student Engagement and Motivation in an Online Course

Authors: Kalpathi Subramanian; Kiran Budhrani

Abstract:

We present a course design model for applying project-based learning to an online undergraduate object oriented systems course. In our model, projects and reflection are central to the curriculum. Our model challenges students through modularized, repetitive project cycles beginning with analysis and design (i.e. using pseudo- code, flowcharts, diagrams) then coding, debugging, testing, and finally, reflection. We analyzed student reflection responses from two semesters to extract major themes and sub-themes, then mapped these to the MUSIC model (eMpowerment, Usefulness, Success, Interest, Caring) to understand our model's influence on student engagement and motivation. We found that a rhythmic project cycle encourages self-regulation in online students to formulate project plans, track their progress, and evaluate their solutions. Online students feel empowered when course projects promote choice, flexibility, creativity, experimentation, and extensions to other applications. Online student success is dependent on the clarity of instructions, course scaffolding, level of challenge, instructor feedback, and opportunities to reflect on personal failure, success, and challenge. Online students are interested in projects that are familiar, real-world, and fun, but expect to be situated in team-based environments. Students appreciate instructors who are caring and accommodating to personal needs. We recommend six salient strategies for improving online course and project design: design a visible, rhythmic structure; set transparent expectations and instructions; encourage design before implementation; connect to real-world applications and tools; experience happy challenges; infuse sustained reflection.

#### A Project-Based Learning Approach for Enhancing Learning Skills and Motivation in Software Engineering

Authors: Beatriz Pérez; Ãngel L. Rubio

Abstract:

Software engineers must be able to manage complex projects, so that skills such as teamwork, leadership or initiative are critical to their successful development. Because of this, it is fundamental that the learning of software engineering as an academic discipline provides solid links between theory and practice. Educational frameworks such as those derived from the European Higher Education Area state that student-centered approaches are a useful tool for achieving these objectives. In this context, we present a project-based learning (PBL) experience report in a software engineering program of a Spanish university. The experience is based on the formation of small heterogeneous teams, which face the initial phases of a software methodology during the development of a project close to a real one. Through a strategy of role rotation and documentation transfer, all students perform different tasks and face different challenges throughout the project. Summative assessment is also adopted, considering not only teacher ratings but also students' peer assessment. The results prove the positive effect of using PBL to improve the training of students in acquiring different skills as future software engineers.

#### The Prevalence of Anxiety and Depression Symptoms among Brazilian Computer Science Students

Authors: Lígia Maria Soares Passos; Christian Murphy; Rita Zhen Chen; Marcos Gonçalves de Santana; Giselle Soares Passos

Abstract:

The mental health of undergraduate students has become a focus of attention for health experts and for society. Several studies have considered students from the Health Sciences, predominantly medical students. Lately, discussions about the mental health of Computer Science (CS) students have become increasingly common.

This paper presents a study about the prevalence of anxiety and depression symptoms among CS students. This study was conducted at Universidade Federal Rural do Rio de Janeiro (UFRRJ), a Brazilian public university. The aim of this study was to verify the prevalence and associated factors of anxiety and depression symptoms among Brazilian CS students.

Students were randomly selected (n=131) from the CS program at UFRRJ, aged between 18 and 30 years. The participants completed a sociodemographic questionnaire and two inventories -- the Beck Anxiety Inventory (BAI) and the Beck Depression Inventory (BDI) -- that were used to measure levels of anxiety and depression symptoms. Statistical analysis was performed using the TIBCO Statistica\textsuperscript{\texttrademark} software (version 13.5).

The prevalence of anxiety and depression symptoms among the students were, respectively, 51.9% and 64.9%, which are higher than previous findings in the Brazilian general population and among medical students. Additionally, the BDI and BAI scores were higher among female students, and we observed statistically significant relationships between BAI, BDI, and sleep quality, among others.

This paper describes our study and findings and addresses ways in which CS students may be able to reduce symptoms of anxiety and depression.

### Session 3A: Novice Programmer Behaviors

#### Exploring Novice Programmers' Homework Practices: Initial Observations of Information Seeking Behaviors

Authors: Silvia Muller; Monica Babe?-Vroman; Mary Emenike; Thu D. Nguyen

Abstract:

There are many factors that contribute to the success of students learning to code. For students in introductory programming classes, one source of complexity is the availability of a wide variety of information sources. In this paper, we report observations of students seeking information when working on programming homework assignments. Our data was collected from a think-aloud protocol embedded into semi-structured, individual interviews with students enrolled in a CS1 course. We analyze our data through the lens of information seeking behavior. We observed students using multiple sources of information, including referring back to course materials and searching for information online, and discussing how they sought help from friends, classmates, and family members. Herein, we discuss implications for teaching and future research based on our initial observations. For example, instructors could consider designing early homework assignments that would prompt students to seek information and follow up this assignment with an in-class discussion about homework strategies. Future research could investigate the mechanisms by which students progress from haphazard to more strategic information seeking behaviors.

#### Sometimes It's Just Sloppiness - Studying Students' Programming Errors and Misconceptions

Authors: Ella Albrecht; Jens Grabowski

Abstract:

Knowledge about students' programming errors is a valuable source to get insights into students deficiencies and misconceptions. In this paper, we use data from an introductory C programming course to identify which errors are often made by students. Previous studies often focused only on syntactic and semantic errors as they can be easily identified by compilers. Studies focusing on logic errors were often restricted to a limited set of concepts or performed for a small set of data. We manually inspect 12371 submission by 280 students and have no restrictions regarding the error types we are looking for. We classify our found errors into six categories: syntactic, conceptual, strategic, sloppiness, misinterpretation, and domain knowledge. Our results show that a big portion of errors made by students is simply caused by sloppiness. But putting sloppiness aside, students seem to have most problems with strategic knowledge, i.e., the problem solving ability. We compare our results to previous studies and provide some implications of our results for future teaching practice.

#### A Comparison of Two Pair Programming Configurations for Upper Elementary Students

Authors: Jennifer Tsan; Jessica Vandenberg; Zarifa Zakaria; Joseph B. Wiggins; Alexander R. Webber; Amanda Bradbury; Collin Lynch; Eric Wiebe; Kristy Boyer

Abstract:

2nd Best Paper, Experience Reports and Tools

As computer science education opportunities for elementary students (grades K-5) are expanding, there is growing interest in using pair programming with these students. However, previous research findings do not fully support its use with younger learners, and some researchers have begun to examine whether introducing a second computer with a shared coding workspace can provide important benefits. This experience report describes a series of classroom activities in the 4th and 5th grades (ages 9-11 years old) with two different pair programming configurations: one-computer pair programming, in which both students share a keyboard, mouse, and monitor; and two-computer pair programming, in which each student has a separate computer but coding workspaces are synchronized over the web. In both cases the students sat next to each other and engaged in face-to-face conversation. We found that students largely preferred two-computer pair programming over one-computer pair programming. We conducted focus groups and transcribed collaborative dialogues to gain more insight into this preference. We learned that students felt more independence in two-computer pair programming, although they struggled with coordinating their edits with their partner. In one-computer pair programming, students reported not wanting to wait for their turn to drive, but feeling as though they communicated more with their partner. Both configurations can be productive for students, but the tradeoffs described in this experience report are important for CS educators and researchers to consider when determining which collaborative configuration to use in each K-5 classroom context.

### Session 3B: CS Instruction

#### CS Student Laptop and Computer Lab Usage as a Factor of Success in Computing Education

Authors: Béatrice Moissinac; Jennifer Parham-Mocello; Robin Pappas

Abstract:

Computing Education Research (CER) implicitly assumes that CS undergraduate students have no barrier to access a learning platform or software package. The assumption that "everyone has access to and uses a device" endangers the validity of CER studies by overlooking a critical element in what students access and how students use computing resources.

First, in this work, we explicitly investigate undergraduate student usage (how often) of computing resources (laptops and computer labs) on a university campus. Furthermore, we investigate whether CS student usage of laptops and computer labs are factors of success in computing education to close a crucial feedback loop for CER and CS educators.

Second, previous studies studying student's technology equipped used qualitative surveys, and lacked a systematic and continuous view of the student population. In this work, we address this shortcoming by developing a method to use operational data sets from the wireless networks and computer labs on campus. Operational data sets provide a systematic and continuous coverage of all students, that is, a student's absence of usage becomes a data point instead of a missing point.

Results indicate that the use of equipment levels may be lower than national average. Nevertheless, there exists a positive correlation between higher frequency of laptop usage and success in computing education.

#### Factors Associated with High-Quality Computer Science Instruction: Data from a Nationally Representative Sample of High School Teachers

Authors: Eric Banilower; Laura Craven

Abstract:

This paper shares results from surveys administered in spring 2018 to a nationally representative sample of nearly 300 U.S. high school computer science teachers. It describes the nature of high school computer science instruction and the extent to which teacher background, classroom factors, and school context predict the type of instruction students experience. Data from the study were analyzed using path modeling-a form of regression analysis that estimates both direct and indirect effects (i.e., through intermediary variables)-to examine relationships between teacher, classroom, and school factors, and the extent to which teachers (1) emphasize reform-oriented instructional objectives (e.g., learning about real-life applications of computer science) and (2) engage students in computer science practices (e.g., recognizing and defining computational problems). Sample findings include that students are most commonly engaged in activities related to testing and refining computational artifacts, but are less often engaged in aspects of computer science related to end users (e.g., create a computational artifact to be used by someone else). The path analysis highlights several factors that are related to greater engagement of students in the computer science practices, including teacher participation in professional development and the use of coherent instructional materials.

#### A Strategy to Enhance Computer Science Teaching Material Using Topic Modelling: Towards Overcoming The Gap Between College And Workplace Skills

Authors: Ingrid Marçal; Rogério Eduardo Garcia; Danilo Eler; Ronaldo Celso Messias Correia

Abstract:

Computer Science teaching materials are biased towards concepts and theoretical aspects. One may consider it difficult to relate concepts to concrete problems. Consequently, it increases the chances of a student not recognizing the relevance of the subject, becoming unmotivated and unprepared to solve practical problems or coping with workplace needs after college. This paper shows the use of social media data as an alternative to minimize the skill gap between what the student learns in college and the skills required in the workplace. The proposed strategy consists of extracting topics from Stack Overflow questions to identify concepts generally unknown or misunderstood and concepts that their practical application represents a challenge. The concepts covered in Stack Overflow questions provide strong cues about how professors and instructors can improve teaching material with useful content for their students, as well as increase their motivation -- since the teaching material becomes clearly related to practical issues in Computer Science. We show, by an example, how to use the proposed strategy to improve teaching material from the generated topics. Also, we demonstrate that the application of topic modeling in Stack Overflow content is promising as a professor support tool to enhance teaching material relevance in Computer Science courses and diminish the college-workplace skill gap.

### Session 3C: CS1/CS2

#### The Practical Details of Building a CS Concept Inventory

Authors: Cynthia Taylor; Michael Clancy; Kevin Webb; Daniel Zingaro; Cynthia Lee; Leo Porter

Abstract:

Concept inventories (CIs) allow researchers and practitioners to measure student conceptual learning within a course or topic area. While they have enabled meaningful pedagogical change in other disciplines, there are relatively few CIs in computer science. In this paper, we report on our experiences as recent developers of a CI for basic data structures. We discuss each step along the route to a CI and offer tips based on what we have learned. We encourage others to create CIs, and we hope that this paper will serve as a practical guide through the process.

#### Reviewing CS1 Materials through a Collaborative Software Engineering Exercise: An Experience Report

Authors: Jessica Schmidt

Abstract:

3rd Best Paper, Experience Reports and Tools

Many courses incorporate exercises at the end of the semester to help students review course topics. In computer science courses, review activities are often small coding problems, interactive games, or review lectures. In this paper, we describe our experience with replacing student-led review presentations with a collaborative software engineering exercise to serve as a review of CS1 programming concepts. The objective of this review exercise is to increase engagement and self-reflection on course topics by engineering a real-world software application to help students review programming concepts and prepare for the subsequent CS1.5 course. During the final two weeks of the course, students spend three class sessions working to complete the comprehensive exercise. During the final class session, each team presents its software. In this paper, we describe the comprehensive exercise, lessons learned from the past eight semesters (over 1,400 students), and suggestions for adopting the comprehensive exercise in other CS1 courses.

#### Exploring the Link Between Prerequisites and Performance in Advanced Data Structures

Authors: Sophia Krause-Levy; Sander Valstar; Leo Porter; William G. Griswold

Abstract:

Recent work has identified a mismatch between instructor expectations of students' mastery of prerequisite course content and their actual ability. This invites the question of why this mismatch exists. We first examined grades in prerequisite courses and found they meaningfully correlated with performance on an assessment testing their knowledge of prerequisite material. In addition, we found neither taking alternatives to the primary identified prerequisites nor the delay between taking prerequisite courses and the follow-on course meaningfully impacts performance. Second, we confirmed that prerequisite course grades are significantly correlated with the grade in the follow-on course-confirming that the grades in the previous courses convey some information about student understanding of those topics. Perhaps surprisingly, we found that grades in courses outside computing were similarly correlated as those courses inside computing, suggesting that underlying factors such as general study skills may be as important as the domain-specific knowledge itself.

### Session 3D: CS + X

#### Abstraction Through Multiple Representations in an Integrated Computational Thinking Environment

Authors: Aakash Gautam; Whitney Bortz; Deborah Tatar

Abstract:

We present reflections based on qualitative analysis of data from the CHEM+C Project which promotes computational thinking (CT) in classrooms through integration with science classes. The curriculum utilizes multiple representations, requiring students to work with physical phenomena, chemical equations, digital simulations, and modifiable code-based representations. Much CT focus on abstraction naturally emphasizes (1) extraction of a set of features from an object or process, and (2) finding commonality between objects and processes. But Rosen encourages us to think about abstraction as also including the production of new concepts or actions. Integrating CT into science offers the possibility of enhancing this aspect of abstraction. Changing the representational affordances available to the students allows them to take their CT thinking beyond learning-to-abstract towards learning-through-abstraction. This perspective moves computation from an internally focused exercise into the expression of valued ideas in a computational medium.

#### Interdisciplinary Computing: Applied Computing for Behavioral and Social Sciences

Authors: Valerie Carr; Morris Jones; Belle Wei

Abstract:

As the digital economy grows, so does the demand for technology-capable workers who have both computing skills and domain expertise. Growing such a workforce is critical to ensuring the nation's competitiveness, according to a recent National Science Board publication. To address this need, faculty from the Colleges of Engineering and Social Sciences at San José State University worked together to create the Applied Computing for Behavioral and Social Sciences minor degree. The minor targets students in majors such as Psychology and Economics, which have a more diverse student population than that of Computer Science or Engineering. The minor, designed with industry input, includes a four-course sequence that focuses on Python and R and includes topics such as data structures, algorithms, data cleaning and management, and data analysis. Our cohort-based program was built specifically for social science students using social science content, helping to foster a sense of community and belongingness among students. The first full cohort of 26 students graduated in Spring 2019, 48% of whom were female and 23% of whom were underrepresented minorities. Our approach of embedding computing education into the social sciences demonstrates a promising model of broadening participation in computing and meeting the nation's increasing demand for interdisciplinary computing workers in the digital age.

#### Interdisciplinary Minor in Bioinformatics: First Results and Outlook

Authors: Natalia Khuri; Wendy Lee; K. Virginia Lehmkuhl-Dakhwe; Miri VanHoven; Sami Khuri

Abstract:

There is a clear demand and need for more and better training in Computer Science for students of different backgrounds and genders. Traditional academic pathways for Computer Science majors and minors may not always be well suited for attracting underrepresented students. With the aim of recruiting and training students from life sciences, over 60 of whom are female, we established an alternative, programming-intensive pathway that is integrated within an interdisciplinary Bioinformatics minor. In this paper, we summarize the development and implementation of this program in a large public university, which can serve as a framework for new interdisciplinary programs in predominantly undergraduate institutions.

### Session 3E: Code Analysis

#### A Study of Keystroke Data in Two Contexts: Written Language and Programming Language Influence Predictability of Learning Outcomes

Authors: John Edwards; Juho Leinonen; Arto Hellas

Abstract:

We study programming process data from two introductory programming courses. Between the course contexts, the programming languages differ, the teaching approaches differ, and the spoken languages differ. In both courses, students' keystroke data -- timestamps and the pressed keys -- are recorded as students work on programming assignments. We study how the keystroke data differs between the contexts, and whether research on predicting course outcomes using keystroke latencies generalizes to other contexts. Our results show that there are differences between the contexts in terms of frequently used keys, which can be partially explained by the differences between the spoken languages and the programming languages. Further, our results suggest that programming process data that can be collected non-intrusive in-situ can be used for predicting course outcomes in multiple contexts. The predictive power, however, varies between contexts possibly because the frequently used keys differ between programming languages and spoken languages. Thus, context-specific fine-tuning of predictive models may be needed.

#### Calibration and Analysis of Source Code Similarity Measures for Verilog Hardware Description Language Projects

Authors: James F. Power; John Waldron

Abstract:

In this paper we report on our experiences during a first-year course on digital logic design using the Verilog hardware description language. As part of the course the students were given a series of take-home assignments, which were then marked using an automated assessment system developed by the authors. During the course the instructor was made aware that a set of solutions had been circulated to the students, and was asked to assess the impact that this had on the assessment regime. In order to answer this question, we examined and implemented a number of approaches to calculating similarity between Verilog programs, and we present the results of that study in this paper. An important feature of this work was ensuring that the measurements used were well-understood, properly calibrated and defensible. We report on the results of this study, applied to a class of 115 students who completed up to 11 projects each.

#### Unlimited Trace Tutor: Learning Code Tracing With Automatically Generated Programs

Authors: Ruixiang Qi; Davide Fossati

Abstract:

Previous research showed that creating specific types of tracing tables helps students learn code tracing, a fundamental skill in computer programming. This paper introduces Unlimited Trace Tutor, the first version of a code tracing tutoring system that can automatically generate tracing problems and create such tracing tables. We conducted a pilot experiment with volunteer students from an introductory level Computer Science course. We found that our software effectively helps student learn tracing "for" loops, "while" loops, and "if" statements. In this paper we describe the system's architecture, our algorithms for generating code and tracing tables, and the promising results of our pilot experiment.

### Session 3F: Computing for Social Good

#### Internationalizing the Student Experience Through Computing for Social Good

Authors: Komal Aheer; Ken Bauer; Cam Macdonell

Abstract:

Information technology has connected our world and its citizens in incredible ways. Despite this connectedness, students are often isolated within the "online bubbles" of their own university, city, or country. Technology provides a great opportunity to connect them to a broader global experience. We have developed and piloted a cross-institution activity as part of an Internationalization at Home (IaH) initiative to expose first year computer science students to the concept of computing for social good in an international context. We explore how differences in culture can influence students' perceptions and approaches to computing for social good. Specifically, we had students from a Mexican and a Canadian university explore how computing for social good could be used to solve issues they faced in their communities. Students participated in surveys to propose and then rank applications for social good. The students also participated in a videoconference discussion with the students from the other school to discuss their choices. Thematic analysis revealed that the students had much more in common with each other than they had differences. Both groups not only focused on similar areas of interest, but they also tended to focus on solving issues with a local scope rather than national or global scope. Despite their cultural differences, the majority students felt they were more similar to their peers of the other culture than they were different.

#### LibreFoodPantry: Developing a Multi-Institutional, Faculty-Led, Humanitarian Free and Open Source Software Community

Authors: Karl R Wurst; Chris Radkowski; Stoney Jackson; Heidi J. C. Ellis; Darci Burdge; Lori Postner

Abstract:

Engaging students in humanitarian free and open source software (HFOSS) projects allows them to gain real-world software development skills while helping society. Participating in an existing HFOSS project, although ripe with learning opportunities, presents a number of hurdles for faculty and students. An alternative to joining an existing HFOSS project community is to participate in a faculty-led HFOSS project. These projects provide the instructor with more control over the learning environment, but often lack an active community outside of the classroom. This paper describes a multi-institutional effort to engage a community of developers in creating humanitarian open source projects to support their on-campus food pantries. Food insecurity on campus has become a national concern and many institutions have, or are starting, food pantries to support the student, staff, and faculty community. Starting a faculty-led HFOSS project involves making decisions not only about the features of the project but also about community norms, tool choices, project development workflow, and inter-institution cooperation. This paper provides an overview of the creation of LibreFoodPantry, a community who is developing a suite of projects that support on-campus food pantries. It describes instances of using LibreFoodPantry's projects in various classroom settings, the lessons learned from these experiences, and the resulting discussions and decisions made by the LibreFoodPantry Coordinating Committee. This process has led to a community dedicated to easing the on-ramp for faculty who want to help their students contribute to an HFOSS project.

#### Merging Sustainability and Technology in the Classroom: An Experience Report

Authors: Giovanni Vincenti; Wolf T. Pecher

Abstract:

Formal education often relies on the students' ability to make connections among different subjects, as they navigate through a curriculum packed with different topics. In order to facilitate the preparation to the workforce, we created a course in "Sustainability and Technology" that was cross-listed in the degrees of Applied Information Technology and Environmental Sustainability. This manuscript documents our efforts in creating an indoor farm by letting students from different majors collaborate on this shared project. The results of our survey support the benefits of letting students collaborate as they approach the end of their undergraduate studies.

### Session 3G: Collaboration

#### Strategies for Implementing In-Class, Active, Programming Assessments: A Multi-Level Model

Authors: Shanon Marie Reckinger; Bryce Hughes

Abstract:

As the enrollment in computer science courses increases, faculty face the challenge of maintaining a standard of academic integrity in out of class programming assignments. One strategy to combat academic misconduct, is to incorporate more frequent in-class assessment. Traditional in-class assessment is often perceived as passive and taking away from valuable class time. Also, more in-class assessment could mean increasing the amount of test anxiety. Therefore, this paper presents results from a study that investigated the effectiveness of active, in-class programming assessment. The study was conducted in an introductory programming course with 93 undergraduate students at a mid-size, public university in the western United States. Three different assessment types were tested: collaborative, cooperative, and individual type assessments. A collaborative assessment is equivalent to pair programming. Cooperative assessments are partner quizzes where students must write their own code and are assessed individually, but may collaborative as much or as little as they like with their partner. Student performance was measured under multiple conditions including type of assessment and partner's active learning preference and factors like GPA, time in the semester, and section were controlled for. The results show that students perform better on both collaborative and cooperative type assessment than they do on individual type assessments. Also, the results show that students perform similarly on collaborative and cooperative assessments, which provides a nice alternative to pair programming for educators. Lastly, results show that most students prefer collaborative assessments over cooperative or individual. The majority of students prefer partner assessment to individual assessment.

#### A Comparison of Peer Instruction and Collaborative Problem Solving in a Computer Architecture Course

Authors: Geoffrey Herman; Sushmita Azad

Abstract:

While substantial research has demonstrated that active learning pedagogies are better for learning than passive lectures, we need to understand the trade-offs between different active learning pedagogies. Computer Architecture at Midwestern University has historically been taught using active lectures, introducing content with a few clicker questions. In Fall 2018 (N = 363), short video lectures were made available to students as a supplemental resource. In Spring 2019, the instructor flipped the course, requiring students to watch the video lectures and complete an assignment before attending class. Two versions of the course were taught concurrently, using the same homework assignments, machine problems, and examinations but with different in-class pedagogies. Version SP19PI (N = 179) was taught using peer instruction. Version SP19CP (N = 73) was taught using collaborative problem solving, organizing students into teams of 3 to work on problems. Students completed surveys that measured their perceptions of time spent on the course, course difficulty, perceptions of stress, and sense of belonging. We compare students' performance on midterm exams and their non-cognitive outcomes to examine the relative effects of these different active learning pedagogies. We find that both flipped offerings (peer instruction and collaborative problem solving) benefited students beyond active lectures. Peer instruction (SP19PI) made learning more efficient. Collaborative instruction (SP19CP) provided greater social support for learning and eliminated gender grade disparities.

#### Exploring the Benefits and Barriers of Using Computational Notebooks for Collaborative Programming Assignments

Authors: Marcel Borowski; Johannes Zagermann; Clemens N. Klokmose; Harald Reiterer; Roman Rädle

Abstract:

Programming assignments in computer science courses are often processed in pairs or groups of students. While working together, students face several shortcomings in today's software: The lack of real-time collaboration capabilities, the setup time of the development environment, and the use of different devices or operating systems can hamper students when working together on assignments. Text processing platforms like Google Docs solve these problems for the writing process of prose text, and computational notebooks like Google Colaboratory for data analysis tasks. However, none of these platforms allows users to implement interactive applications. We deployed a web-based literate programming system for three months during an introductory course on application development to explore how collaborative programming practices unfold and how the structure of computational notebooks affect the development. During the course, pairs of students solved weekly programming assignments. We analyzed data from weekly questionnaires, three focus groups with students and teaching assistants, and keystroke-level log data to facilitate the understanding of the subtleties of collaborative programming with computational notebooks. Findings reveal that there are distinct collaboration patterns; the preferred collaboration pattern varied between pairs and even varied within pairs over the course of three months. Recognizing these distinct collaboration patterns can help to design future computational notebooks for collaborative programming assignments.

### Session 4A: Learning B

#### Qualitative Analyses of Movements Between Task-level and Code-level Thinking of Novice Programmers

Authors: Francisco Castro; Kathi Fisler

Abstract:

Cognitive theories of how programmers produce code suggest that novices' approaches are primarily driven by the retrieval of previously-learned plans. These plans can be high-level, focusing on task decomposition and composition, or low-level, focusing on code implementations. These theories, however, do not describe how novices move between high-level tasks and low-level code, especially when faced with novel problems. Understanding these transitions can help concretely tease out why and where novices struggle and how they use their knowledge of plans and design techniques when they get stuck.

We studied this by conducting think-alouds with CS1 students at two universities as they solved multi-task programming problems with unfamiliar components. Our analysis paid particular attention to a series of design techniques that the students had been explicitly taught in their respective courses. We identified patterns of high- and low-level thinking that relate to students' success on the problems, and propose a concrete framework of high- and low-level work that summarizes the transitions that we observed.

#### Challenges with Learning to Program and Problem Solve: An Analysis of Student Online Discussions

Authors: Paul Piwek; Simon Savage

Abstract:

Students who study problem solving and programming (in a language such as Python) at University level encounter a range of challenges, from low-level issues with code that won't compile to misconceptions about the threshold concepts and skills. The current study complements existing findings on errors, misconceptions, difficulties and challenges obtained from students after-the-fact through instruments such as questionnaires and interviews. In our study, we analysed the posts from students of a large cohort (~1500) of first-year University distance learning students to an online `Python help forum' - recording issues and discussions as the students encountered specific challenges. Posts were coded in terms of topics, and subsequently thematically grouped into Python-related, problem solving/generic programming related, and module specific. We discuss the set of topics and rank these in terms of the number of forum discussions in which they occur (as a proxy for their prevalence). The top challenges we identified concern student understanding and use of a mix of programming environments (in particular, Python IDLE for offline programming and CodeRunner for programming quizzes) and code fragment problems. Apart from these, Python-specific topics include, among others, collections, functions, error messages, iteration, outputting results, indentation, variables and imports. We believe that the results provide a good insight into the challenges that students encounter as they learn to program. In future work we intend to study the discussions in further detail in terms of theories of conceptual change.

#### A Study on Student Performance Evaluation using Discussion Board Networks

Authors: Urvashi Desai; Vijayalakshmi Ramasamy; James D. Kiper

Abstract:

Node-based social network analysis (SNA) techniques can be used to investigate the significance of actors that play central roles in social networks where the nodes represent people, teams or stakeholders and the links represent the communication, information exchange or collaboration between these nodes (actors). This research investigates how collaborative problem-solving can help in students' learning process. We analyzed the discussion board data collected from online student discussions on Canvas, a Learning Management System (LMS), in a CS1 course of a medium-sized US University. The discussion topics were classified as classroom experiences/learning, question/answers, opinions, and comments and were used to represent the patterns of interactions in the student discussion networks. Node-based network measures were then applied to unravel the students' interaction patterns to gain insights on students' progress. The textual analysis helps find the most challenging/debated topics in a particular course, analyze the leadership and team-based qualities of a group of students, and analyze patterns and trends in female student participation. The experimental analysis reveals that participation in online discussion forums has a positive impact on the students' grades; the study of interaction patterns exhibit similar insights. In conclusion, this research study validates that the analysis of structured discussions can provide useful insights into changes in student collaboration patterns over time and students' sense of belongingness for pedagogical benefits.

### Session 4B: Elementary School Experiences

#### Future-Proofing Kiwi Kids Through the Use of Digital Technology

Authors: Veronica Liesaputra; Guillermo Ramirez-Prado; Bashar Barmada; Lei Song

Abstract:

The common approach for introducing middle school and high school students to computational thinking and engineering is through programming or robotics activities that were designed without much social context. Therefore, one of the challenges such outreach programs pose is further stereotyping of students who are interested in computer science, by presenting them as people who ought to be hyper-intelligent and very much focused on computers with no social skills.

Realizing the above, and wanting to attract a diverse audience to study computer science, we developed a one-day workshop which showcases the interdisciplinary nature of computer science and the various ways computing can help students discover and solve societal problems. We chose multiculturalism, cyber security and air pollution as the context of our computational thinking activities.

This paper describes our experiences in developing and teaching the workshops that runs three to five times a year, hosted either in our institute or at local schools. Regardless of students' previous STEM experiences and socioeconomic statuses, both students and teachers reported that the knowledge gained enabled them to either create a computer program or a robot and also led to a better understanding of how computers can be used in daily life. In other words, the workshop successfully opened the minds and perspectives of the middle school and high school students on computer science. Additionally, over time, we also observed an increased number of students and schools participating in such workshops.

#### Patterns in Elementary-Age Student Responses to Personalized & Generic Code Comprehension Questions

Authors: Jean Salac; Qi Jin; Zipporah Klain; Saranya Turimella; Max White; Diana Franklin

Abstract:

The CS community has struggled to assess student learning at the K-8 level, with techniques ranging from one-on-one interviews to written assessments. While scalable, automated techniques exist for analyzing student code, a scalable method for assessing student comprehension of their own code has remained elusive. This study is a first step in bridging the gap between the knowledge gained from interviews and the time efficiency and scalability of written assessments and automated analysis. The goal of this study is to understand how student answers on various types of questions differ depending on whether they are being asked about their own code or generic code. We find that while there were no statistically-significant differences in overall scores, questions about generic and personalized code of comparable complexity are far from equivalent. Our qualitative analyses revealed interesting patterns in student responses, inviting further research into this assessment technique. In particular, students answered differently from students with generic code when presented with individual blocks from their code taken out of context and placed into different code snippets, and students answered in a way that demonstrates a functional, instead of structural, understanding on Explain in Plain English (EiPE) questions.

#### Comparing TUIs and GUIs for Primary School Programming

Authors: Abrar Almjally; Kate Howland; Judith Good

Abstract:

There is considerable interest in using tangible user interfaces (TUIs) to support teaching children programming, but evidence for the benefits is mixed, and their deployment in school environments presents more challenges than graphical user interfaces (GUIs). This study investigates the effect of GUIs and TUIs on learning outcomes, attitudes toward computing, and reported enjoyment in a computer-programming activity with primary-school students aged 6-7 in Saudi Arabia. Forty-two students engaged in a 45-minute learning activity using either a TUI or GUI programming environment. The study used a between-groups design, and quantitative data were collected, including pre-test and post-test results, and ratings on attitudinal and enjoyment surveys. Learning gains were significantly higher for the GUI group than the TUI group. However, post-activity increases in reported attitude toward computing were significantly higher for the TUI group. There was no difference in activity enjoyment scores, which were high for both groups.

### Session 4C: BPC

#### Applying NCWIT Protocol to Broaden Participation in Computing: A Case Study of CS@Mines

Authors: Tracy Camp; Christine Liebe; Michelle Slattery

Abstract:

Best Paper, Experience Reports and Tools

The Department of Computer Science (CS@Mines) at Colorado School of Mines (Mines) was founded in 2016 when the Department of Electrical Engineering and Computer Science (CS) split into separate departments. As a result, CS faculty, who had worked for years to broaden participation in computing without departmental lead- ership support, were able to become more strategic in their efforts. CS@Mines faculty, staff, and students now engage in well-defined recruitment, retention, and evaluation strategies, which includes K-12 outreach programs, flexible CS major and minor tracks, a near-peer mentoring program, scholarship programs, and contin- ual evaluation. Ten years ago, the CS degree program at Mines had 157 majors, 17 women (10.8%), and 12 students from underrepre- sented groups in computing (7.6%). As of Fall 2019, CS@Mines has 679 majors, 146 women (21.5%), and 132 students from underrep- resented groups (20.2%). Although the concentrated effort focused on increasing the number of women majors, the data clearly shows an increase in students from underrepresented groups as well. The changes achieved by CS@Mines are noteworthy considering (1) only 30% of the students at Mines are women, (2) only 17% of the students at Mines are from underrepresented groups in computing, and (3) women and underrepresented groups enrolled in undergraduate CS programs are predominantly not at parity with their respective populations in the United States. CS@Mines achieved positive results by applying the Undergraduate Systemic Change Model developed by the National Center for Women & Information Technology. In this paper, we present CS@Mines as a case study for positive change and discuss the strategies CS@Mines has enacted (74 of the recommended 81).

#### Experience of Administering Our First S-STEM Program to Broaden Participation in Computer Science

Authors: An-I Andy Wang; David Whalley; Zhenghao Zhang; Gary Tyson

Abstract:

This paper documents the findings of our analysis of the implementation of our six-year NSF S-STEM scholarship program. One major finding was that, for underrepresented students to major in computer science, knowing the major existed and understanding the nature of the program were the most important factors. Also, the academic support system and hands-on nature of the major had a significant impact on scholarship recipients' persistence in the major. Evidence demonstrated that scholarship recipients had a 10%+ higher year-to-year persistence rate from their freshmen to sophomore year than that of all computer science students of the same entering classes. For all computer science students, college computer science major GPAs were not strongly correlated with their high school GPAs, financial need, or ACT math scores. This paper also presents lessons learned and resulting recommendations for future new scholarship administrators, as our lessons can likely be applied to other grants that recruit and deal with underrepresented groups.

#### Does AP CS Principles Broaden Participation in Computing? An Analysis of APCSA and APCSP Participants

Authors: Linda Sax; Kaitlin N. S. Newhouse; Joanna Goode; Max Skorodinsky; Tomoko M. Nakajima; Michelle Sendowski

Abstract:

A major attempt to broaden participation in computer science has centered on the design and development of a new high school Advanced Placement (AP) course, AP Computer Science Principles (CSP). This course was created to intentionally engage a wider and more diverse group of students in learning about computing than those who had historically enrolled in programming-focused AP Computer Science "A" (CSA). After several years in the pilot phase, the course was officially offered by the College Board in 2016-17. This paper uses nationwide Freshman Survey data to examine the demographics and characteristics of students who took either CSA, the new CSP course, or both courses by the time they entered college in Fall 2017. Using crosstabs, z-tests, and one-way ANOVA, this study uncovered the similarities and differences between students who had participated in one or both of these courses. Our findings suggest that while students who took only CSP were more diverse than those who took only CSA, they exhibited less computing confidence and less interest in computing majors and tech careers. This study suggests that while CSP may be recruiting more females and racially diverse students into the course, this course alone may not serve as a direct pipeline into computing majors and careers.

### Session 4D: Rubrics and Evaluation

#### Adaptive Rubrics

Authors: Marco Carmosino; Mia Minnes

Abstract:

Grading is a notoriously difficult and time-consuming part of teaching. For open-ended programming, mathematical, or design problems, assigning consistent scores and giving useful feedback can be very challenging. Large classes compound this difficulty. Adding TAs to the team can help parallelize the process but may impede grading consistency and quality. We present an adaptive rubric creation and application process to enable high-quality responses to student work, at scale. This process uses exploratory data analysis to discover common patterns in student responses to a problem, then tailors a rubric and feedback to address these patterns. Our method is supported by current grading tools, which allow calculation of the simple population-level statistics we need to extract meaningful features from a corpus of student work. In this case study, we describe using adaptive rubrics for a discrete math class for CS majors: the grading team found that this process produced concrete and transparent justifications of student scores and that it facilitated conversations around grading that were grounded in course learning objectives and values.

#### A Large-scale Evaluation of a Rubric for the Automatic Assessment of Algorithms and Programming Concepts

Authors: Nathalia Alves; Christiane Gresse von Wangenheim; Jean Hauck; Adriano Ferreti Borgatto

Abstract:

As computing education makes its way into schools, there is still little research on how to assess the learning of algorithms and programming concepts as a central topic. Furthermore, in order to ensure valid instructional feedback, an important concern is the reliability and construct validity of an assessment model. Therefore, this work presents a large-scale evaluation of the CodeMaster rubric for the performance-based assessment of algorithms and programming concepts by analyzing software artifacts created by students as part of complex, open-ended learning activities. The assessment is automated through a web-based tool that performs a static analysis of the source code of App Inventor projects. Based on 88,812 projects from the App Inventor Gallery, we statistically analyzed the reliability and construct validity of the rubric. Results indicate that the rubric can be regarded as reliable (Cronbach's alpha = 0.84). With respect to construct validity, there also exists an indication of convergent validity based on the results of a correlation and factor analysis. This indicates that the rubric can be used for a valid assessment of algorithm and programming concepts of App Inventor programs as part of a comprehensive assessment completed by other assessment methods. The results can guide the improvement of assessment models, as well as support the decision on the application of the rubric in order to support computing education in K-12.

#### A Validated Scoring Rubric for Explain-in-Plain-English Questions

Authors: Binglin Chen; Sushmita Azad; Rajarshi Haldar; Matthew West; Craig Zilles

Abstract:

Previous research has identified the ability to read code and understand its high-level purpose as an important developmental skill that is harder to do (for a given piece of code) than executing code in one's head for a given input ("code tracing"), but easier to do than writing the code. Prior work involving code reading ("Explain in plain English") problems, have used a scoring rubric inspired by the SOLO taxonomy, but we found it difficult to employ because it didn't adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity.

In this paper, we describe a 7-point rubric that we developed for scoring student responses to "Explain in plain English" questions, and we validate this rubric through four means. First, we find that the scale can be reliably applied with with a median Krippendorff's alpha (inter-rater reliability) of 0.775. Second, we report on an experiment to assess the validity of our scale. Third, we find that a survey consisting of 12 code reading questions had a high internal consistency (Cronbach's alpha = 0.954). Last, we find that our scores for code reading questions in a large enrollment (N = 452) data structures course are correlated (Pearson's R = 0.555) to code writing performance to a similar degree as found in previous work.

### Session 4E: Faculty

#### Faculty Adoption of CS Education Innovations: Exploring Continued Use

Authors: Christopher Hovey; Lecia Barker

Abstract:

Studies on faculty adoption typically focus on the awareness and trial stages: how instructors find out about a new teaching practice and why they decide to try it. While this knowledge is important, reform is unlikely to occur if innovations are only used experimentally; we need to understand why faculty sustain their use. To that end, this study draws on data from a two-phase project in the U.S. to examine why computer science (CS) faculty continue or discontinue use of a practice after the initial trial. In the first phase, we interviewed and observed 66 CS faculty from various institutional settings to explore theory-driven themes. From these results, in the second phase, we collected survey data from 821 faculty at 595 institutions. In this paper, we briefly discuss qualitative results, and then use quantitative data to model what impact the following factors have on sustaining use: achieved benefits for students' performance and their satisfaction in CS, the perception of usefulness to the instructor, student feedback, and ease of use. Results indicate that benefits to students' performance are paramount in predicting continued use. We also explore why faculty abandon a practice, finding that the decision often relates to not achieving desired outcomes and, in some cases, students not making a good faith effort to do their part. We observed that the latter experience can engender negative beliefs about students-that they have, at best, a passive attitude toward their learning. Implications for encouraging sustained usage of innovative teaching techniques are discussed.

#### Faculty Views on the Goals of an Undergraduate CS Education and the Academia-Industry Gap

Authors: Sander Valstar; Sophia Krause-Levy; Alexandra Macedo; William G. Griswold; Leo Porter

Abstract:

Previous work has found that recent computer science graduates often experience difficulty transitioning into their new roles in industry due to a significant gap between their academic experiences and industry's expectations. Although multiple studies have identified the views of students and members of industry on the value of a CS degree as preparation for industry, the faculty perspective on this topic remains unclear. Understanding these views could shed light on why the academia-industry gap has persisted despite the attention. This study identified faculty views on the goals of an undergraduate education and a CS major, focusing on preparation for careers in industry. In order to identify a spectrum of faculty views, we interviewed 14 faculty from a variety of backgrounds across three institutions. A phenomenographic analysis of the transcripts reveals that many faculty believe that industry preparation is an important programmatic goal, yet they encounter significant resource obstacles to achieving that goal.

#### Linking Faculty Attitudes to Pedagogical Choices: Student-Centered Teaching in Introductory Computing Classes

Authors: Christopher Hovey; Kathleen Lehman; Tiffani Riggers-Piehl

Abstract:

Research suggests using student-centered practices in the classroom is a key component of attracting and retaining diverse students. To better understand the link between attitudes toward students and learning and the usage of specific teaching strategies, we analyze survey responses from 54 faculty who teach introductory computer science (CS) courses from 15 U.S. colleges and universities participating in BRAID. Using principal component analysis, we scale responses to 10 attitudinal questions into four dimensions: rugged individualism ("learning and success are the individual student's responsibility"), challenging work ("the pace and workload in CS is hard"); a developmental orientation to learning ("students need individual attention in a non-competitive environment"); and capable students ("all students can do well in my class"). We then test these scales on four types of teaching: three student-centered approaches-collaborative learning approaches, discourse activities, and student-led learning-and one traditional approach, lecturing. Results indicate that a developmental orientation predicts the use of discourse activities and student-led practices, but not collaborative learning approaches. Rugged individualism is associated with frequent lecturing. None of our scales predict the use of collaborative learning approaches, and neither attitudes for "challenging work" nor "capable students" predict any of the pedagogical approaches in our study. We examine differences by certain faculty characteristics and discuss the ramifications of these results for promoting more widespread adoption of student-centered teaching.

### Session 4F: Computing Education Research

#### A Gap Analysis of Statistical Data Reporting in K-12 Computing Education Research: Recommendations for Improvement

Authors: Monica M. McGill; Adrienne Decker

Abstract:

The quality of reporting of experimental results in computing education literature has been previously shown to be less than rigorous. In this study, we first examined research standards set forth by four organizations: American Psychology Association (APA), American Educational Research Association (AERA), What Works Clearinghouse (WWC), and the CONsolidated Standards of Reporting Trials (CONSORT). We selected the most important data standards based on their prominence across all four and the most typical study designs in computing education research. We then examined 76 articles designated as quantitative research studies (K-12) published in ten venues (2012-2018) to determine whether the reporting in these articles met these five standards. Findings indicate that only 48% of these articles report effect size and even fewer (11%) report confidence intervals and levels. We found that reported data did not meet the standard that data should be "reported in a way that the reader could construct effect-size estimates and confidence intervals beyond those supplied in the paper". Additionally, authors used existing instruments less than a quarter of the time (24%) and used instruments with evidence of reliability and validity less than half of the time (39%). We conclude with recommendations for those in the K-12 computing education research community to consider when reporting statistical data in future work so that we can increase the level of rigorous reporting in this growing field.

#### Three Metrics of Success for High School CSforAll Initiatives: Demographic Patterns from 2003 to 2019 on Advanced Placement Computer Science Exams

Authors: Kip Lim; Colleen Lewis

Abstract:

There has been an expansion of computer science (CS) in high schools in the USA. In most cases, initiatives seeking to expand high school CS offerings are not focused simply on increasing the number of students enrolled, but are focused on broadening participation in computing (BPC). Ideally we can evaluate and replicate effective BPC, or CSforAll, initiatives at the high school level. However, analyses of CS expansion frequently focus on the national landscape, despite the decentralized nature of public education in the USA. State-level analyses, when present, often do not take into account the demographics of high school students in the state, which are also changing over time. We propose three metrics for evaluating the impact of state-level initiatives to broaden participation in computing at the high school level. These metrics and our data set can be helpful benchmarks for evaluating high school CSforAll initiatives.

#### A Longitudinal Analysis of K-12 Computing Education Research in the United States: Implications and Recommendations for Change

Authors: Bishakha Upadhyaya; Monica M. McGill; Adrienne Decker

Abstract:

The availability of computer science education in primary and secondary schools in the United States has grown rapidly over the last decade. Computing education research in K-12 has been increasing as well. In this study, we conducted a longitudinal analysis of seven years of data (2012-2018), curated from over 500 articles across 10 publication venues to identify trends in K-12 computing education research such as geographic location and curriculum and concepts taught. The data shows a decrease in the number of studies covering K-12 students receiving computing education even while there is an increase in the number of states adopting and implementing standards. The number of different concepts being researched is increasing, potentially reflecting the growth in what is being taught in the classroom. Demographic data is underreported (e.g., socio-economic status (SES) and disabilities of participants) which could directly limit generalizability of the studies to different learners as well as the ability to replicate and compare studies. We conclude with recommendations for how to better position this work for others trying to use the results to guide their efforts in creating standards or adopting techniques into their classrooms.

### Session 4G: Database

#### Worked Examples, Cognitive Load, and Exam Assessments in a Senior Database Course

Authors: Edwin M. Knorr

Abstract:

This paper deals with an upper-year course on internals of relational database systems. It is an experience report about switching from a teaching model using traditional lecturing and homework assignments, to a partially flipped model that replaces regular homework with an abundance of worked examples and more frequent full-length exams. We turn to cognitive load theory to help address the challenges of problem-solving in complex topics such as query evaluation and optimization. Worked examples are used to try to reduce students' cognitive load. Students' confidence levels were captured through surveys. After seeing and practicing many worked examples, students claim confidence in mastering the learning goals. We use Bloom's Dichotomous Key to create a Bloom's Taxonomy weighted index to track the difficulty of our exams over the years, both before and after the switch. Although final exam grades have remained relatively flat, our exams have been getting more difficult, suggesting improved learning, even though we have not seen a significant increase in performance on transfer questions. Principal Component Analysis is used to gain insight into our final exam, showing that a large fraction of the exam questions are needed to account for most of the variance in students' scores.

#### Mapping the SQL Learning Process in Introductory Database Courses

Authors: Andrew Migler; Alexander Dekhtyar

Abstract:

The Structured Query Language (SQL) is powerful, prevalent across many problem domains, and challenging to master. Sophisticated information requests require programmers to set aside familiar procedural and functional modes of thought. Instead, programmers must learn how to apply unfamiliar set-based relational manipulation rules. This paper analyzes the student learning process in an introductory database course. We do this using detailed analysis of problem-solving attempts by 114 students related to 116 SQL lab exercises assigned over a five-week period. We measure student success rates in mastering these SQL concepts, as well as effort expended by students in solving information retrieval problems.

#### Group Exams as Learning Tools: Evidence from an Undergraduate Database Course

Authors: Jalal Kawash; Tamer Jarada; Mohammad Moshirpour

Abstract:

Peer-instruction has been shown to be an effective method to support learning. We exploit a form of peer-instruction in an undergraduate course on databases, where students take an exam in teams. Instantly after attempting the exam individually without any immediate feedback, students re-take the exam as a group utilizing immediate feedback instruments. These instruments permit the students to tackle each multiple-choice question several times until a correct answer is uncovered. Our thesis is that this approach provides the students with an opportunity to learn from their mistakes, whether committed individually or as a group, while promoting individual student deep learning. We support this thesis by analyzing data collected from 125 students, 5 group exams, and other assessment instruments, including a final exam.

### Session 5A: Software Engineering

#### What Agile Processes Should We Use in Software Engineering Course Projects?

Authors: An Ju; Adnan Hemani; Yannis Dimitriadis; Fox Armando

Abstract:

While project-based software engineering courses aim to provide learning opportunities grounded in professional processes, it is not always possible to replicate every process in classrooms due to course constraints. Previous studies observed how students react to various processes and gave retroactive recommendations. In this study, we instead combine a field study on professional Agile (eXtreme Programming, XP) teams and an established team process taxonomy to proactively select team processes to incorporate in a project-based software engineering course. With collected knowledge from the field study, we choose three XP processes to augment the design of a mature software engineering project course. We choose processes that are 1) considered important by professionals, and 2) complete with respect to coverage of the taxonomy's main categories. We then compare the augmented course design with the original design in a case study. Our results suggest that 1) even without extra resources, adding these new processes does not interfere with learning opportunities for XP processes previously existing in the course design; 2) student teams experience similar benefits from these new processes as professional teams do, and students appreciate the usefulness and value of the processes. In other words, our approach allows instructors to make conscious choices of XP processes that improve student learning outcomes while exposing students to a more complete set of processes and thus preparing them better for professional careers. Course designers with limited resources are encouraged to use our methodology to evaluate and improve the designs of their own project-based courses.

#### Assessing Individual Contributions to Software Engineering Projects with Git Logs and User Stories

Authors: Kevin Buffardi

Abstract:

Software Engineering courses often incorporate large-scale projects with collaboration between students working in teams. However, it is difficult to objectively assess individual students when their projects are a product of collaborative efforts. This study explores measurements of individuals' contributions to their respective teams.

I analyzed ten Software Engineering team projects (n=42) and evaluations of individual contributions using automated evaluation of the version control system history (Git logs) and user stories completed on their project management (Kanban) boards. Unique insights from meta-data within the Git history and Kanban board user stories reveal complicated relationships between these measurements and traditional assessments, such as peer review and subjective instructor evaluation. From the results, I suggest supplementing and validating traditional assessments with insights from individuals' commit history and user story contributions.

#### Teaching Autonomous Systems at 1/10th-scale: Design of the F1/10 Racecar, Simulators and Curriculum

Authors: Abhijeet Agnihotri; Matthew O'Kelly; Houssam Abbas; Rahul Mangharam

Abstract:

2nd Best Paper, Curricula Initiatives

### Session 8C: Learning Tools

#### User-Centered Design of a Mobile Java Practice App: A Comparison of Question Formats

Authors: Mohona Ahmed; Kimberly Ying; Kristy Boyer

Abstract:

Learning computer science presents many challenges to students, and providing resources for meaningful practice is recognized as a way to support rigorous learning and diverse student participation. At the same time, mobile phones are increasingly ubiquitous, creating an underutilized opportunity for practice outside of traditional methods. This experience report presents a user-centered approach to designing a practice app for introductory Java. We investigated user preferences through a series of small studies, first conducting think-aloud sessions and focus groups, and finally conducting a usability study comparing two prototype versions. The initial studies suggested how to leverage the affordances of small screens, ruling out free-response practice problems in favor of either fill-in-the-blank (FB) or multiple-choice (MC) questions. The comparison study revealed statistically significant differences in students' survey responses: (1) usability scores were significantly higher for the MC version than the FB version; (2) students reported significantly greater satisfaction and desire to learn for the MC version; and (3) students reported enjoying and being more comfortable with the MC version compared to the FB version. We contextualize this observation within related research on question formats. Takeaways from this experience report can provide guidance on designing mobile applications that give students opportunities for meaningful practice.

#### How Important is Immersion for Learning in Computer Science Replugged Games?

Authors: Andreas Dengel

Abstract:

By following the idea of not using computers at all, Computer Science Unplugged has set the course for many interactive, social, and hands-on activities dealing with concepts and problems of Computer Science Education. Through recent developments in immersive media, technologies like Virtual and Augmented Reality could enhance such activities or even enable new ones. When providing immersive educational media that induce a sense of presence in the virtual environment, the illusion of unmediated learning experience can be delivered. Hence, the concept of "Computer Science Replugged" can benefit from the affordances that Computer Science Unplugged thrives on while facilitating or enabling activities that might be impossible, dangerous, or expensive to carry out in reality. This paper presents three concepts from Computer Science Education that have been modeled as 3-D immersive educational virtual environments: components of a computer, asymmetric encryption/decryption, and finite state machines. To get a first impression of the effectiveness of these approaches and in order to determine the importance of the level of immersion for the learning process, a study with 78 participants was conducted in which the software was tested on different devices. All activities were found to be significantly effective with regards to the pre- and post-tests. When analyzing these results on the basis of comparing the least immersive setting (laptop) with the most immersive setting (head-mounted-display), the findings indicate different effects with effect sizes between beta = -.17 and beta</i> = .41. This raises two questions: "Which topics from Computer Science Education can benefit from immersive technology?" and "What are the opportunities and challenges of the didactical design of Computer Science Replugged activities?" that have to be adressed in further research. </p>

#### Student Adoption and Perceptions of a Web Integrated Development Environment: An Experience Report

Authors: Martin Valez; Michael Yen; Mathew Le; Zhendong Su; Mohammad Amin Alipour

Abstract:

Students often spend a considerable amount of time and effort installing and configuring programming tools and environments. This can frustrate, and distract them from more important learning objectives, particularly in introductory programming courses. A web-based integrated development environment can serve as a low-threshold, ready-to-use programming environment, and reduce the time and effort needed to start practicing programming.

In this paper, we report our experience of developing and deploying a web-based integrated development environment (web IDE) as an optional tool at a large public university that has been in use for over several years in various programming courses.

We conducted a survey to understand students' perceptions toward the web IDE and usage of its features. Using the data from the survey, we explored potential correlations between student demographic and behavioral traits in adoption of the web IDE. The results of the survey suggest that around half of the students use the IDE often or very often. We also discovered that the likelihood of adoption of the IDE decreases as students to move to upper classes. In this paper, we also describe broader lessons for educators and researchers.

### Session 8D: Online

#### Improving Student Learning Outcomes in Online Courses: An Investigation Into the Effects of Multiple Teaching Modalities

Authors: Julio César Bahamón; Audrey Rorrer

Abstract:

We investigated learning outcomes within an undergraduate C Programming course that is taught in multiple modalities: in-person, online and blended. Because this course has been taught by the same instructor, using the same scaffolding activities, materials and approaches, we were uniquely positioned to conduct a quasi-experimental study of learning outcomes between courses and within students. The overarching goal was to glean knowledge and implications about assessment practices for undergraduate courses that are taught in multiple modalities. The objectives of our research are primarily to discern what differential impacts, if any, are found between the in-person and the online course delivery. We aimed to discover learning outcome patterns among the students who participate in these modalities. Findings from this study provide valuable information for undergraduate Computer Science programs by identifying any differential learning outcomes that students experience between in-person and online course instruction. The research questions addressed by the study were as follows: 1) What impact does modality have on student learning outcomes? 2) What patterns are discernable across student groups? 3) What relationship is there between final course grades and assignment module learning outcomes? In earlier work, we were surprised that no significant differences were obtained between course modality. While this was an encouraging finding, we believed that further data collection and analysis were needed, before making general conclusions about the two modalities. This paper reports on our efforts to collect additional data, while considering additional variables, such as instructor, multiple modalities, and online course design approaches.

#### Incorporating Active Learning Strategies and Instructor Presence into an Online Discrete Mathematics Class

Authors: Sandy Irani; Kameryn Denaro

Abstract:

Online education offers an attractive alternative to face-to-face classes by providing flexibility to students and efficiencies for educational institutions. Leveraging online technology has the potential to help computer science departments offer a quality educational experience in the face of burgeoning enrollments. However, effective online course design is critical to student satisfaction and learning outcomes. In this paper, we describe the experience of converting a large face-to-face course in Discrete Mathematics to an online format. Particular care was taken to incorporate active learning strategies, such as clicker questions and interactive discussions, in order to enhance student engagement. We describe ways in which we cultivated an active instructor presence in the course through carefully designed pre-recorded videos, online video conferencing, and participation in Piazza, an online social learning platform.

In-class tests were specifically designed to provide a meaningful comparison of learning outcomes between a face-to-face and online offering of the course taught by the same instructor. The results indicate that there is no loss in student performance in the online course, even after accounting for demographic and academic differences between the students enrolled in the two courses. There is also no significant difference in performance for "at-risk" students. End-of-quarter student evaluations show a high level of student satisfaction with the online format, especially in regards to the opportunities to have questions answered and the positive presence of the instructor in the course.

#### Who Benefits? Positive Learner Outcomes from Behavioral Analytics of Online Lecture Video Viewing Using ClassTranscribe

Authors: Lawrence Angrave; Zhilin Zhang; Genevieve Henricks; Chirantan Mahipal

Abstract:

Lecture material of a sophomore large-enrollment (N=271) system programming 15-week class was delivered solely online using a new video-based web platform. The platform provided accurate accessible transcriptions and captioning plus a custom text-searchable interface to rapidly find relevant video moments from the entire course. The system logged student searching and viewing behaviors as fine-grained web browser interaction events including full-screen-switching, loss-of-focus, incremental searching events, and continued-video-watching events with the latter at 15-second granularity. Student learning behaviors and findings from three research questions are presented using individual-level performance and interaction data. Firstly, we report on learning outcomes from alternative learning paths that arise from the course's application of Universal Design for Learning principles. Secondly, final exam performance was equal or better to prior semesters that utilized traditional in-person live lectures. Thirdly, learning outcomes of low and high performing students were analyzed independently by grouping students into four quartiles based on their non-final-exam course performance of programming assignments and quizzes. We introduce and justify an empirically-defined qualification threshold for sufficient video minutes viewed for each group. In all quartiles, students who watched an above-threshold of video minutes improved their in-group final exam performance (ranging from +6 % to +14%) with the largest gain for the lowest-performing quartile. The improvement was similar in magnitude for all groups when expressed as a fraction of unrewarded final exam points. Overall, the study presents and evaluates how learner use of online video using ClassTranscribe predicts course performance and positive learning outcomes.

### Session 8E: Data Science B

#### Collective Development of Large Scale Data Science Products via Modularized Assignments: An Experience Report

Authors: Bhavya -; Assma Boughoula; Aaron Green; ChengXiang Zhai

Abstract:

Many universities are offering data science (DS) courses to fulfill the growing demands for skilled DS practitioners. Assignments and projects are essential parts of the DS curriculum as they enable students to gain hands-on experience in real-world DS tasks. However, most current assignments and projects are lacking in at least one of two ways: 1) they do not comprehensively teach all the steps involved in the complete workflow of DS projects; 2) students work on separate problems individually or in small teams, limiting the scale and impact of their solutions. To overcome these limitations, we envision novel synergistic modular assignments where a large number of students work collectively on all the tasks required to develop a large-scale DS product. The resulting product can be continuously improved with students' contributions every semester.

We report our experience with developing and deploying such an assignment in an Information Retrieval course. Through the assignment, students collectively developed a search engine for finding expert faculty specializing in a given field. This shows the utility of such assignments both for teaching useful DS skills and driving innovation and research. We share useful lessons for other instructors to adopt similar assignments for their DS courses.

#### Cross-Disciplinary Faculty Development in Data Science Principles for Classroom Integration

Authors: Brandeis Marshall; Susan Geier

Abstract:

Data science in practice leverages the expertise in computer science, mathematics and statistics with applications in any field using data. The formalization of data science educational and pedagogical strategic remain in their infancy. College faculty from various disciplines are tasked with designing and delivering data science instruction without the formal knowledge of how data science principles are executed in practice. We call this the data science instruction gap. Also, these faculties are implementing their discipline's standard pedagogical strategies to their understanding of data science. In this paper, we present our cross-disciplinary instructional program model designed to narrow the data science instruction gap for faculty. It is designed to scaffold college faculties' data science learning to support their discipline-specific data science instruction. We provide individualized and group-based support structures to instill data science principles and transition them from learners to educators in data science. Lastly, we share our model's impact on and value to faculty as well as make recommendations for model adoption.

#### Expanding Undergraduate Exposure to Computer Science Subfields: Resources and Lessons from a Hands-on Computational Biology Workshop

Authors: Layla Oesper; Anya Vostinar

Abstract:

Computational biology is an exciting and ever-widening interdisciplinary field. Expanding the participation of undergraduate students in this field will help to inspire and train the next generation of scientists necessary to support this growing area. However, students at smaller institutions, such as those focused on undergraduate education, may not have access to courses related to or even faculty interested in computational biology. Providing more opportunities for such undergraduate students to be exposed to computational biology, or other subfields within computer science, will be important for ensuring these students are included in the pipeline of scientists contributing to these diverse fields. To this end, we hosted a computational biology workshop that brought together undergraduate students from three different liberal arts colleges. The goal of the workshop was to provide an introduction to how computer science can be used to help answer important problems in biology. A diverse set of six faculty members from different institutions each created and taught a hands-on module as an introduction to a different area of computational biology at the workshop. We describe how we went about organizing this undergraduate workshop, summarize the workshop materials that are freely available, and discuss the outcomes and lessons learned from the workshop. We further propose that the workshop structure used is adaptable to other subfields of computer science. Workshop materials available at the workshop website: https://sites.google.com/carleton.edu/compbioworkshop2018/home.

### Session 8F: K12 Teaching

#### Teaching Elementary Computer Science through Universal Design for Learning

Authors: Maya Israel; Gakyung Jeong; Meg Ray; Todd Alan Lash

Abstract:

Given the academic diversity of today's classrooms, elementary teachers engaged in computer science (CS) and computational thinking (CT) instruction must create CS/CT experiences that are accessible and engaging to a broad range of learners, including those with disabilities. One method of developing inclusive instructional experiences is through the Universal Design for Learning (UDL) framework, wherein teachers proactively design instruction for the broadest range of learners. Doing so may be challenging as elementary teachers may not be familiar with the UDL framework or may not have experience with applying UDL within CS/CT instruction. The purpose of this qualitative study was to investigate how four elementary teachers provided UDL-based instruction to academically diverse learners during CS/CT instruction. Teachers received professional development and instructional coaching related to UDL within CS/CT education. Data included teachers' lesson plans, coaching logs, and teacher interviews which were qualitatively analyzed and triangulated. Data revealed that teachers generally addressed all three UDL principles, with an emphasis on two of the principles (multiple means of engagement and multiple means of representing content) above the third principle (multiple means of action and expression). They focused on breaking tasks into steps, emphasizing student choice, and presenting information in multiple ways. Findings revealed nuanced implementation differences among the teachers as well.

#### Design Studios for K-12 Computing Education

Authors: Megan Venn-Wycherley; Christine Bennett; Ahmed Kharrufa

Abstract:

Following the decline of pupil engagement in compulsory computing education (K-12) in the UK, advocates have called for further research into computing-specific pedagogies. Aiming for an improvement in pupil engagement, the subsequent report explores the viability of the applied Design Studio approach in K-12 computing education. We report on two case studies involving two 12-week curricula co-developed and delivered between researchers and teachers for four Year 8 classes in a secondary school in England. Common to both case studies were the design brief, desk crit, design review, presentation and portfolio elements proposed by Koutsabasis et al, with the key difference being a change in scope and structure of the challenge presented to pupils. Our findings demonstrate the need for a well-structured, personalizable and challenge-driven Design Studio approach, which showed evidence of improved pupil resilience and confidence, while engaging reluctant pupils. We conclude with a proposed model for applying Design Studios in K-12 computing education, to support educators aiming to adopt project-driven computing pedagogy in the classroom.

#### Impacts of Creating Smart Everyday Objects on Young Female Students' Programming Skills and Attitudes

Authors: Mazyar Seraj; Eva Katterfeldt; Serge Autexier; Rolf Drechsler

Abstract:

In computer programming education, learning to program tangible objects has become a common way to introduce programming to young students. In an effort to address this intervention, scientific research has been done on the effectiveness of using tangible hardware platforms such as robots and wearable products to teach basic programming concepts to children. However, there is a lack of research on how young students' attitudes and programming skills are influenced over time, when they learn to program tangible objects and make them smart. In this paper, we investigate the impacts of using a tangible everyday object and making it smart on young female students' attitudes towards programming and the acquisition of basic programming skills. During a 4-day non-formal programming workshop with 12 6th grade students, they were introduced to basic programming concepts, and learned how to apply them to turn a houseplant into a smart object. In a pilot study, we employed a block-based programming environment and analyzed the students' trajectories of attitudes towards programming and performance based on repeated open-ended qualitative questionnaires and programming questions throughout the workshop. The results show that all students had high confidence regarding programming skills, regardless of creating smart objects. Furthermore, it indicates that experienced students highly valued the programming of tangible everyday objects compared with inexperienced students. The findings of this work contribute to our understanding of how making tangible everyday objects smart can support the development of a positive attitude and keep up of interest throughout a programming workshop among girls.

### Session 8G: Graduate Programs

#### Enrollment Motivations in an Online Graduate CS Program: Trends & Gender- and Age-Based Differences

Authors: Alex Duncan; Bobbie Eicher; David Joyner

Abstract:

Demand for CS education has risen, leading to numerous new programs, such as the rise of affordable online degrees. Research shows these programs meet an otherwise untapped audience of working professionals seeking graduate level CS education. In this study, we examine the motivations for enrollment among students in one such online MSCS program. Based responses to an open ended question, we develop a typology of motivations, including goals (e.g. career transition), opportunities (e.g. enrolling without taking time off work), and assurances that their goals will be met (e..g the program's accreditation). We then issue a closed survey question to a new group of students to further explore these motivations. In this paper, we discuss both aggregate and demographic trends in motivations, including the different motivations of men and women and what they imply about the program's impact on the gender divide in computing. We also examine older students' tendency towards intrinsic motivation to pursue an MSCS degree.

#### An MS in CS for non-CS Majors: Moving to Increase Diversity of Thought and Demographics in CS

Authors: Carla Brodley; Megan Barry; Aidan Connell; Catherine Gill; Ian Gorton; Benjamin Hescott; Bryan Lackaye; Cynthia LuBien; Leena Razzaq; Amit Shesh; Tiffani Williams; Andrea Danyluk

Abstract:

We have created, piloted and are growing the Align program, a Master of Science in Computer Science (MS in CS) for post-secondary graduates who did not major in CS. Our goal is to create a pathway to CS for all students, with particular attention to women and underrepresented minorities. Indeed, women represent 57% and underrepresented minorities represent 25% of all bachelor's recipients in the U.S., but only 19.5% and 12.6% of CS graduates, respectively. If we can fill this opportunity gap, we will satisfy a major economic need and address an issue of social equity and inclusion. In this paper, we present our "Bridge" curriculum, which is a two-semester preparation for students to then join the traditional MS in CS students in master's-level classes. We describe co-curricular activities designed to help students succeed in the program. We present our empirical findings around enrollment, demographics, retention and job outcomes. Among our findings is that Align students outperform our traditional MS in CS students in grade point average. To date we have graduated 137 students and 827 are enrolled.

#### A Constructivist Redesign of a Graduate-level CS Course to Address Content Obsolescence and Student Motivation

Authors: Lorenzo Angeli; Juan José Jara Laconich; Maurizio Marchese

Abstract:

The last decade has seen a rising popularity of active learning methodologies in Computer Science, empowering students and developing their soft skills as well as their technical knowledge. In parallel, the speed of technological obsolescence also increased, creating challenges for teachers to keep their course content fresh and up to date. In this paper, we present a constructivist redesign of a Graduate-level laboratory course in Web Service Design and Engineering that leverages latent pockets of student knowledge to tackle these challenges through Learning by Teaching. We illustrate how such redesign was planned, deployed and evaluated, highlighting the guiding role of teachers in the process and discussing how this approach was able to solve the problem of keeping content updated while broadening both content and tools students were exposed to. Furthermore, we will discuss how the additional motivation stemming from their empowerment allowed students not only to perform more work compared to a lecture-based implementation, but also to perceive it in the end as a lesser load.