Description and Goals

In this paper, I describe a hypothetical technological framework for sound synthesis, which works as the other half of a human-machine cocreative system, grounded at least in part on a simulation of the cognitive capacity for analogy-making. My goal is not to design a system that generates music integrally or autonomously—i.e., where the machine appears to act by its own accord, producing finished or almost finished works or segments of works—but to better understand analogy and how it can be computationally exploited to foster and enhance human creativity, in particular in the context of my own artistic practice and the aesthetic values that it entails.

Analogy at the Core of Musical Thought

Through analogy we compare things that are different, but share relevant commonalities, allowing us to project cognitively resonant structures between them and gain new functional insights. As the “fuel and fire of thinking” (Hofstadter and Sander 2013), analogy is central to a wide range of human abilities, ubiquitous in everyday thought, and determinant for our worldly experience. It is thus unsurprising that analogy shows up prominently in music. Participating in the musical phenomenon involves the cognition of sound-pattern formation and the mapping of gestural-temporal processes, which shape how we make, listen, think in or about music, feel and move through it, individually and collectively. Knowledge from a variety of domains is carried over, or projected, into sound, thus constituting our musical experience. Such projection is the characteristic mark of analogy.

Notwithstanding the centrality of analogical processes, musical studies have only recently started to examine their conceptual implications. Some accounts focus on correspondences through recurrence “within music,” such as thematic/formal roles (Bourne 2015), where the schematic repetition or transformation of a pattern in successive musical passages gives shape to “chains of analogy” (Kielian-Gilbert 1990). As analogical comparisons drive the processes of conceptualization and abstraction, analogy lies at the core, for example, of the concept of motives, through which we understand each new instance of a musical pattern by comparing it with other ones that we previously encountered, noting their shared structure despite the superficial dissimilarities. Analogy is also implicated in the cognition of metric groupings, or the exposition and recapitulation of a sonata form, or for that matter in the chorus/verse recurrence of a pop song. Other accounts focus on structural commonalities between different parameters, e.g., pitch and time (Bar-Yosef 2007; Eitan and Granot 2007). Furthermore, the suggestion that music works fundamentally as a “sonic analog for dynamic processes” (Zbikowski 2017) connects music with emotion, gesture, dance, or words. These connections mount to the view that our conceptual system is prominently metaphoric, i.e. constrained by the features of the human body and worldly experience (Lakoff and Johnson 1980), and structured by sensorimotor schematic patterns (Johnson 2007). Such a perspective, together with the analogy-like theory of conceptual blending (Fauconnier and Turner 2002), prompted emergent frameworks on how music is conceptualized (Brower 2000; Hatten 1995; Larson 2012; Saslaw 1996; Spitzer 2004; Zbikowski 2002).

Since, in these cases, music appears to be “standing for” a distinct reality, Zbikowski (2017) highlights that music makes use of a unique form of reference—“analogical reference”—which can be understood in terms of Peircean semiotics, and particularly in terms of the concept of the icon. Zbikowski observes that, as Peirce divided the icon into image, diagram, and metaphor, sonic analogs can be traced in a continuum between those categories, where sounds that more clearly mimic an actual audible event are positioned closer to image, and the sonic analogs for nonsonic dynamic processes closer to metaphor. Symbolic reference, by contrast, while predominant in language, is residual in music, being relegated to instances where a musical utterance is conventionally correlated with a specific referent, as it notably happens with the culturally shared associations that constitute the object of topic theory.

Structure-Mapping and Analogy in Artificial Intelligence

The computational modeling of analogy enjoys a rich history. Understandably, if the capacity for analogy is such a critical mark of intelligence, it follows that it must be somehow introduced in artificial intelligence systems. On the other hand, research on analogy as a cognitive process arose and was developed contemporaneously with the general perspective positing that human reasoning can be understood through its implementation in computer programs. The technical approaches for artificial analogy-making have followed the broader trends in the field of artificial intelligence, from older (but still promising) symbolic methods, which are based on the manipulation of symbols representing the knowledge for the base and target domains, to more recent deep learning techniques, as well as hybrid architectures. Reviewing several of these approaches, Mitchell (2021) concludes that despite the extensive efforts, which remain as active as ever, “no current AI system is anywhere close to a capability of forming humanlike abstractions or analogies,” while at the same time, such advances will be key for continued progress going forth from current state-of-the-art artificial intelligence models.

According to structure-mapping theory (Gentner 1983, 1989; Gentner and Smith 2013; Gentner et al. 2001), developed for the last four decades and now in some ways the classic, empirically validated framework for analogy, we are biased toward mapping relational structures, and preferably systems of mutually connected higher-order relations, and not so much object properties or attributes—this preference is called the systematicity principle. This is why we find the analogy between a house and a nest (same functional relationships) more compelling than the one between a planet and a ball (same shape)—see Figure 1.

Audio Synthesis Through Analogy-Driven Mappings

The idea of applying cognitively-resonant domain-general computational models of analogy to music, or integrating implementations such as the SME to sound generation tasks, remains largely unexplored. Some tentative music-related approaches (Eppe et al. 2018; Zacharakis et al. 2021) have instead followed the conceptual blending framework (Fauconnier and Turner 2002), which describes a very similar high-level cognitive process where elements and relations from two or more domains are compared, but conceptualizing their combination as a fusion (blend) into a new integrated mental space. The integration network model is meticulously specified and, as it is apt for formalization, has been an attractive framework for computational approaches. On the other hand, conceptual blending is targeted to the creation of hybrids, and thus it’s less flexible than more general models of analogy.

Otherwise, I find the tenets of structure-mapping particularly apt for the domain of music. Music is highly relational, at the very least because of its intrinsic temporality. What is the value of a single sound event, if it’s not taken in relationship with past and future ones? Besides, higher-order relations are manifest in the pervasiveness of conceptualizations that organize sound in hierarchies, processual configurations, or cause-effect chains.

In this context, I envision a representation of the sound domain that proceeds from segmenting an audio stream into multiple very short sonic tokens, quantified according to music information retrieval metrics, which are correlated temporally with image schematic patterns such as containment, source-path-goal, interruption, self-similarity, or pendulum. Such information is then stored in a corpus database. From here, the SME probes and acts upon hypothetical cross- or intra-domain mappings. Cross-domain mappings are made possible by having the non-sonic domain categorized through the same common image schemas. Music-to-music mappings would have another audio stream as the base for the analogy.

The creation of new sounds is accomplished through a kind of concatenative synthesis (Schwarz 2004)—a method of generating audio by selecting and assembling small sonic units from a large database of sound sources. Typically, the selection and assemblage are performed by attempting to match quantitative physical, perceptual, or statistical features (e.g., pitch, spectral centroid, average amplitude, tempo) of the sources. Such feature-matching depends on a specification of criteria for the similarity between sonic units. In the various kinds of domain comparison that were contrasted above, this kind of similarity would approximate “mere appearance,” since it deals predominantly with collections of object properties. In the analogy-driven setup that I propose, however, mappings would be established not according to the similarity of surface features, but according to the degree of isomorphism in relational structures.

Thus, mapped sonic units would not necessarily sound similar; instead, the resulting audio stream would exhibit deeper structural commonalities perceived as convincing, compelling, and surprising, despite the superficial differences—just like the analogies that we rely upon in our day-to-day life.

In conclusion, I believe that this strategy leads to a machine-generating but human-steerable framework for producing novel timbres and sonic textures. One that, by being grounded in the cognitive capacity for analogy, exhibits a degree of creativity still lacking in artificial intelligence systems, and whose expected glitches, non-linearities, and incoherences could be artistically useful in music-making.