Cross-evaluation of metrics to estimate the significance of creative works

In a world overflowing with creative works, it is useful to be able to filter out the unimportant works so that the significant ones can be identified and thereby absorbed. An automated method could provide an objective approach for evaluating the significance of works on a universal scale. However, there have been few attempts at creating such a measure, and there are few “ground truths” for validating the effectiveness of potential metrics for significance. For movies, the US Library of Congress’s National Film Registry (NFR) contains American films that are “culturally, historically, or aesthetically significant” as chosen through a careful evaluation and deliberation process. By analyzing a network of citations between 15,425 United States-produced films procured from the Internet Movie Database (IMDb), we obtain several automated metrics for significance. The best of these metrics is able to indicate a film’s presence in the NFR at least as well or better than metrics based on aggregated expert opinions or large population surveys. Importantly, automated metrics can easily be applied to older films for which no other rating may be available. Our results may have implications for the evaluation of other creative works such as scientific research.For many types of creative works—including films, novels, plays, poems, paintings, and scientific research—there are important efforts for identifying which creations are of the highest quality and to honor their creators, including the Oscars, the Pulitzer Prize, and the Nobel. Unfortunately, these distinctions recognize only a small number of creators and sometimes generate more controversy than consensus. The reason is that one of the challenges associated with measuring the intrinsic quality of a creative work is how to formally define “quality.”In statistical modeling, this problem is typically addressed by positing the existence of latent (hidden) variables, which are unmeasurable but can be inferred from the values of other, measurable variables (1). For creative works, we presume there exists a latent variable, which we call “significance.” Significance can be thought of as the lasting importance of a creative work. Significant works stand the test of time through novel ideas or breakthrough discoveries that change the landscape of a field or culture. Under this perspective, what is usually called “quality” is not the actual value of the latent variable, but an individual’s or group’s estimation of that value. Not surprisingly, the subjective evaluation of the unmeasurable true significance of the work is controversial, dependent on the historical moment, and very much “in the eye of the beholder.”Alternative methods for estimating the significance of a creative work fall under the labels of “impact” and “influence.” Impact may be defined as the overall effect of a creative work on an individual, industry, or society at large, and it can be measured as sales, downloads, media mentions, or other possible means. However, in many cases, impact may be a poor proxy for significance. For example, Duck Soup (2) is generally considered to be the Marx Brothers’ greatest film, but it was a financial disappointment for Paramount Pictures in 1933 (3). Influence may be defined as the extent to which a creative work is a source of inspiration for later works. Although this perspective provides a more nuanced estimation of significance, it is also more difficult to measure. For example, Ingmar Bergman’s influence on later film directors is undebatable (4, 5), but not easily quantified. Despite different strengths and limitations, any quantitative approaches that result in an adequate estimation of significance should be strongly correlated when evaluated over a large corpus of creative works.By definition, the latent variable for a creative work is inaccessible. However, for the medium of films—which will be the focus of this work—there is in fact as close to a measurement of the latent variable as one could hope for. In 1988, the US Government established the US National Film Preservation Board (NFPB) as part of the Library of Congress (6). The NFPB is tasked with selecting films deemed “culturally, historically, or aesthetically significant” for preservation in the National Film Registry (NFR). The NFR currently comprises 625 films “of enduring importance to American culture” (7). The careful evaluation and deliberation involved in the selection process each year, and the requirement of films being at least 10 y old to be eligible for induction, demonstrates the NFPB’s true commitment to identifying films of significance.Presence in the NFR is a binary variable as no distinctions are made between inducted films. This means that, although it can function as a “ground truth” for significances above a threshold value, it cannot discern the comparative significance of films. One of the goals of this study is to determine whether there are metrics that can accurately estimate film significance over a range of numerical values and for a large number of films. To this end, we investigate proxies of film quality, impact, or influence as potential measures of significance.One can identify three main classes of approaches for estimating the significance of films: expert opinions, wisdom of the crowd, and automated methods. Expert opinions tend to measure the subjective quality of a film, whereas wisdom-of-the-crowd approaches tend to produce metrics that measure impact or popularity through broad-based surveys. Ideally, we can obtain an automated method that can measure influence. However, the best-known automated methods for films pertain to economic impact, such as the opening weekend or total box office gross. More recently, researchers and film industry professionals have evaluated films using electronic measures, such as Twitter mentions (8) and frequency of Wikipedia edits (9), but these may also be better indicators of impact or popularity. For an automated, objective measure that pertains to a film’s influence, we turn to scientific works for an appropriate analog.The network formed by citations from scientific literature is at the center of much research (10–12). Although some of the research on the scientific citation network aims to answer questions on citations between academic fields (13) or sex bias in academia (14), much work seeks to determine who is “winning” at science (15). Researchers have identified numerous metrics that are said to determine which paper (16), researcher (17), or journal (18) is the best, most significant, or most influential. These metrics range from the simple, such as total number of citations (19), to the complex, such as PageRank (20). The scientific citation network provides large quantities of data to analyze and dissect (12, 15, 21). If it were not for the expectation that researchers cite relevant literature, these metrics and indeed this avenue of study would not exist.Like scientists, artists are often influenced or inspired by prior works. However, unlike researchers, artists are typically not obligated to cite the influences on their work. If data identifying citations between creative works could be made or obtained, we then could apply citation-based analyses to develop an objective metric for estimating the significance of a given work. As it happens, such data now exists. The Internet Movie Database (IMDb) (www.imdb.com) holds the largest digital collection of metadata on films, television programs, and other visual media. For each film listed in IMDb, there are multiple sections, from information about the cast and crew to critic reviews and notable quotes. Nestled among the deluge of metadata for each film is a section titled “connections,” which contains a list of references and links to and from other films (Fig. 1). By analyzing this citation network obtained from user-edited data, we can investigate the suitability of metrics to estimate film significance based on the spread of influence in the world of motion pictures.Open in a separate windowFig. 1.Subgraph of film connections network. Films are ordered chronologically, based on year of release, from bottom to top (not to scale). A connection between two films exists if a sequence, sentence, character, or other part of the referenced film has been adopted, used, or imitated in the referencing film. For example, there is a connection from 1987’s Raising Arizona (22) to 1981’s The Evil Dead (23) because the main characters of both films drive an Oldsmobile Delta 88. Values represent the time lag of the connection, measured in years.