2021 Results

Participants

Eleven teams from around the world (representing 20 different countries and 6 continents) participated in the data challenge in 2021.

About QE Methods

There are many ways to conduct quantitative ethnography (QE) research, but QE analyses have five interdependent elements that distinguish them from other approaches to modeling rich qualitative data.

1. Grounded Analysis

QE researchers begin with a thorough qualitative analysis of their data (or sample of their data). This facilitates informed interpretation of the data, hypothesis generation, and identification of constructs that may explain the cognitive, behavioral, or interactive phenomena of interest.

2. Data Coding

To locate constructs in their data, researchers develop a system for determining which constructs are present in which parts of the data. That system can be as simple as a human rater reading each line to determine whether a given construct is in evidence.

3. Automated Coding Process

When researchers work with large datasets, it is not possible to make coding decisions manually for every excerpt. To identify key constructs in data at scale, researchers often develop automated coding algorithms.

4. Coding Validation

Regardless of the coding process used, researchers need to validate their codes to establish that the coding decisions are consistent and accurate. This requires, at a minimum, establishing sufficient agreement between two independent human raters (for manual coding) or between two independent human raters and the automated coding algorithms (for automated coding).

5. Model Specification

Once a dataset has been coded for the constructs of interest, researchers develop models of those codes to conduct exploratory or confirmatory analyses. This involves selecting a technique that can model the relationships among codes found in the qualitative analysis, and choosing variables that define the unit of analysis and other model parameters. Here, a particular emphasis in QE is aligning the quantitative model with the pattern observed in the qualitative analysis, so that the quantitative model can warrant that the grounded pattern observed is consistent throughout the data.

6. Qualitative Regrounding (Loop Closing)

Finally, researchers confirm that a result from their model is meaningful through a process of qualitative regrounding. That is, they confirm that the outputs and explanatory properties of a model are aligned with the findings of their original grounded analysis, thus closing the interpretive loop.

Findings

Because the Data Challenge lasted only a week, teams did not complete analyses to the point of being ready for peer review. Any findings should thus be treated with appropriate caution.

To aid in the interpretation of results, teams indicated which of the steps of their QE analysis were completed, in progress, or planned.