1. Sensing security

The first paper in this bundle starts from the observation that across fields as diverse as defence, police and migration control, algorithmic data processing is collapsed under a practice of ‘sensing’. How can we as researchers of security relate to such a description of technology? The paper documents a year of collective discussion between me and four other scholars, who share a concern with the non-neutral ways of ‘sensing’ in, and ‘making sense’ of security practices.

Speaking of sensing, my co-authors and I argue, implicitly mobilises a belief that these technologies merely augment the human sensorium, improving on its sensitivity and reach. However, sensors are not mere prosthetics external to the human body, providing access to new sources of information, but they “are changing the subjects of experience as well” (Gabrys, 2016: 22, see also p.65). As such, we argue, the composition of sensing technologies matters for how threats are perceived and interpreted, while shaping the security practices of which they are part.

The sensor itself is not just a measurement instrument, a mere input point of data, but produces its signal by means of computational techniques. What is called a sensor can often be unpacked as camera+computer. Such computational sensors enact a sequence of ‘transductions’ (Mackenzie, 2002), each transforming, converting and interpreting the sensor input to produce an output signal. At each step, these transductions articulate and rearticulate measured realities (see also Steyerl, 2014). Sensing practices deploying camera+computer=sensors reconfigure relations of power between surveiller and surveilled.

At stake in ‘vision’ of ‘computer vision’, is thus not the relation to the human eye, but the onto-epistemological formations of data-processing such technologies gives rise to. Hence, the collective paper advocates, critical study of security should describe the cascades of transformations taking place to dissociate sensing practices from the human senses and open the monolithic sensor up to alternative compositions.

2. Diagramming algorithmic politics

The second paper this dissertation bundles, turns to sociotechnical practices in which ‘computer vision’ and ‘security’ feed into each other, to further inquire how such technologies enable practitioners to see, sense and surveill. The paper grew out of a desire to map the use of computer vision technologies in security practices, in order to flesh out relevant research questions. But how is one map algorithmic practices? Early on in our research, my co-authors and I attempted such a mapping by neatly categorising the practices we encountered – databases, companies, algorithms, etc (see Ragazzi et al., 2021). However, categorising complex algorithmic devices proved difficult (for example, databases are not singular systems, they run ‘algorithms’; which algorithms then would deserve their own entry, which do not?). We kept changing the schema of the collected data: refining categories, adding attributes and relations. Yet, we felt ever-more conflicted: were we not reproducing the rigid systems of classification our research intended to inquire? Were we not mobilising the exact same aesthetics of data visualisation that we had become so sceptical of? (see Fuller and Weizman, 2021; Drucker, 2011; Plájás et al., 2020 documents our conversations; see also Ragazzi, 2023). In retrospect, the questions we had then – the reluctance to pin down practices into marked categories, and the unwillingness to disentangle our own position from the aesthetics of research – touch upon this thesis’ methodological proposition: re-modulation inquires the tools by which we get to know algorithms.

The resulting paper proposes an exploratory methodological device, a time-based diagramming tool. Diagramming is an open-ended, associative and processual approach that complements interviews with a real-time drawing of diagrams³. The paper maps out how the boundaries of algorithmic systems are drawn and negotiated by practitioners, and how entities solidify and stabilise as they circulate between sites of development and deployment. By examining the drawings of the interviews, the paper sketches out a rhizome of interrelated processes and politics, through which my co-authors and me draw five pathways, each indicative of shifting politics of algorithmic security.

Two of the shifts highlighted in this section are of particular relevance for the computational assessment of movement, and the filtering of suspicious subjects. The first concerns a shift in algorithmic perception, from what one might call a photographic vision to a cinematic vision. A photographic vision is about knowing the individual, it relies on the inscription of supposed ‘indelible’ truths in supposedly static biological features of individuals (faces, fingerprints, retinas). Of course, such a photographic logic of algorithmic perception can be applied to with sequences of snapshots that are taken over a period of time. However, as increases in processing power significantly decrease the interval between individual sensor readings, a different interpretative logic becomes possible. This cinematic vision operationalises the variability of life, thereby constructing suspicion as time-based, ephemeral and changing.

A second shift highlighted in this section makes apparent how an algorithm’s failure to make the right prediction is not a defect of algorithmic systems, but a central characteristic. In ‘machine learning’, the most prominent technique for contemporary computer vision, a model converges by means of an error metric. This metric, due to the complexity and ambiguity of life, can never reach zero. If anything, an error-free model is suspect, considered ‘overfitted’ on the training data, and as failed to generalise on the patterns that it had been presented with. Consequently, in computational sense making, reality can never be known for certain. The inevitable presence of errors in the predictions of algorithmic systems means security practitioners can not blindly rely on these results. This uncertainty influences how these devices are integrated in security practices.

3. Prediction of the now

The third text of this dissertation builds on both insights – the cinematic algorithmic vision, and the inevitable uncertainty of prediction – as it addresses a deceptively simple question: what is movement in algorithmic terms? The paper turns to DeepSORT, a prominent algorithmic technique for ‘multi-object tracking’ that transduces immobile, frame-by-frame, detections into moving subjects. By means of a re-modulation of the tracker, the paper technographically traces the logics at play, and contends there is no observation of movement without its assessment.

By examining concepts from signal processing and information theory that movement tracking relies on, I suggest these technologies require rethinking the subject of security. The political effects of algorithmic movement tracking cannot be accounted for by existing debates on technology in security. In these debates, the political effects of technology are either analysed within a linear temporal framework of immediacy – in which new technologies make subjects move faster or slower (e.g. Leese and Pollozek, 2023; Walters, 2017) – or they pertain to the calculative techniques by which the future is anticipated in terms of potential harm (e.g. Amoore and De Goede, 2008; Aradau et al., 2008; Lyon and Wood, 2021; Massumi, 2010). However, algorithmically accounting for movement not only increases the velocity of a still subject, but places its perception in a different temporal logic.

The section upholds that with the algorithmic tracking of movement, the ‘now’ is predictively enacted through an anticipatory model of normality. With the prediction of the now, the subject under surveillance is no longer assessed in terms of its similarity to a ‘risky other’, or in terms of potential future harm. Instead the other emerges by logics of anticipation and surprise in the temporal variability between algorithmic prediction and measurement. This section draws on formulas from signal processing, which are to resolve the fundamental variability and uncertainty of measuring reality, to formulate an alternative role for the subject under surveillance. It concludes that, when the subject is governed by means of a model of normality, it becomes complicit to the assessment of others.

4. Perplexity

This reconfiguring of the moving subject of surveillance is further developed in an interactive art installation that is part of this dissertation, and which is complemented by short essay: Perplexity. Perplexity re-modulates an algorithmic anomaly detector. Such algorithmic systems track bodies as they move through space, recurrently predicting its future movements. Suspicion emerges when a subject deviates from the predicted trajectory.

Consisting of LiDAR sensors, a computer, and laser projectors, the installation re-appropriates these generative surveillance algorithms. Like an anomaly tracker, Perplexity tracks people passing through space, and captures their trajectories. The captured tracks are used to continuously train a model to forecast the movements of others. In this re-modulation, the forecasted trajectories are drawn right in front of the passerby’s feet as pathways of light. Thereby, the installation creates an interaction between the pedestrian and the predictive algorithm. The participant is not only interpreting the object, but its movements inform the unfolding of the piece for whoever passes the space at a later time.

Re-modulating an anomaly detector, and reflecting on my own position towards surveillance as I traverse public space, this section brings to bear the role of the surveilled as co-constutitive of surveillance. Descriptions of surveillance, algorithmic or not, tend to describe a one-way flow of information from the surveilled to the surveillor. Such descriptions render the distinction between surveiller and surveilled as a binary. Yet thinking through security practices in terms of signal/noise, and anticipation/surprise, upsets this binary. Security practices govern movement by measuring conformity (see also Huysmans, 2023), and maintaining normality. When we consider the production of that normality, which is so explicitly modelled in algorithmic surveillance, it becomes apparent that any observed subject contributes to the predictive mechanics by which the other’s movements are anticipated and thus assessed.

To an observer, most of our movements are static⁴. This noise is irrelevant to their task. Suspicion, then, is established over time, by gradually filtering a signal from everything and everyone else in the ether. Consequently, we should not discount the role the surveilled subject plays in the surveillance of others. For standing out is a standing out against others, and thus the other stands out against you.

Engaging with Literature on Sensing Security

In this collective discussion piece, the notion of “sensing security” provides a framework for re-orienting our analyses to explore the mediated ways in which insecurities are made perceptible, whilst also illuminating how security practices are enabled and constrained by them. One of the most important insights to emerge from CSS is how the perception and normalization of security threats takes shape in different practices (Bigo, 2002; Huysmans, 2006). These practices of securitization are characterized by deployment of an increased technological sophistication—ranging from surveillance systems, biometric scanners, to algorithmic prediction tools—that not only facilitate but significantly shape how security actors sense and make sense of security threats (Amicelle et al., 2015; Bellanova et al., 2021; Hoijtink and Leese, 2019a; Ruckenstein, 2023). In this collective discussion piece, we feel indebted to these approaches, including the focus on the materiality and agency of objects of security (Salter, 2015; Walters, 2002). Building on this scholarship, we, therefore, frame sensing security not as a purely mechanical or calculative practice, but as “a delicate interplay between humans, artefacts, and discourses” (Klimburg-Witjes et al., 2021: 28). More specifically, we argue that the value of “sensing security” lies in directing analytical attention toward the technologies and techniques that mediate both the perceptions and actions of security practitioners (Gabrys, 2016; Verbeek, 2005: 111–117).

In this collective discussion piece, we therefore analyze first how sensory technologies play a role in shaping how security is perceived. The existing literature in IPS and CSS already offers valuable insights. For instance, research on biometric border control demonstrates the work of detecting and verifying the identities of mobile populations (Glouftsios and Casaglia, 2023). Biometric technologies are designed to capture and display physical characteristics or parts of the human body that are not perceptible to the naked eye (Van Der Ploeg, 1999). However, how these sensory technologies depict their sensory input shapes how the body is perceived, interpreted, and acted upon and how they perceive border crossers (Valkenburg and Van Der Ploeg, 2015).

In addition, our contributions also shed light on the manner in which these technological mediators enable and constrain the actions of security practitioners. They not only impact how security actors engage with sensory technological mediations when perceiving and interpreting risks and threats, but also how they render these uncertainties actionable and governable (Amoore and Raley, 2017). This facet has been shown in the literature to be of particular pertinence when considering militarized sensing technologies, wherein strong ideals exist for seamlessly integrating the processes of perception and annihilation (Bousquet, 2018; Richardson, 2024). For instance, Gregory (2011) shows not only how remote drone operators engage with the video feeds from the aerial platform when perceiving and interpreting the battleground, but also what are influences of these “new visibilities” on military actions. He thereby argues that this sensing practice “produce[s] a special kind of intimacy that consistently privileges the view of the hunter-killer” with “implications [that] are far more deadly” (Gregory, 2011: 193). These sensory technologies thus shape not only what one perceives as suspicious, risky or a threat, but also who is the subject of surveillance and targeting. Although drone warfare might be experienced as a neutral intermediary by those using it—a more “precise,” “surgical,” or “objective” form of war conduct—it carries a form of sovereign power to decide the exception (Shah, 2012; Wilcox, 2017).

Engaging with the existing literature on the subject, the starting point for our understanding of “sensing security” is therefore that the knowledge about insecurities is not derived from direct perception; rather, what is experienced is always a mediated practice. Already, the “linguistic turn” in security studies taught us that there is no direct experience of something without the mediation through various discourses regarding security. For instance, the way in which border control, terrorism, drugs, organized crime, migration, asylum or the environment are perceived and treated politically depends on how they are represented discursively as dangerous, threatening, alarming and so on (Buzan et al., 1998). Building on these insights, more recent scholarship has focused on how human senses mediate perception (see also Bergson, 2012). Notably, seeing is often singled out, not only for being bound up with (Western) conceptions of truth and knowledge (Brighenti, 2007), but also a practice of surveillance (Amoore, 2009; Tazzioli and Walters, 2016). In addition, smell has been shown to be a form of embodied sensing capable of apprehending potential threat and enmity (McSorley, 2020). Moreover, Weitzel (2018: 421) has looked at “the central roles that sound, hearing, and voice” play in security practices. What we take from these different accounts is that there is not one dominant mode of sensing and perceiving a world that is to be secured, but there are multiple ways of sensing security.

However, as noted above, scholarship in IPS and CSS has increasingly de-centered the human subject and shifted its attention towards how technologies mediate sense-making. Perception is not limited to humans—nor to living beings, for that matter (Bourne et al., 2015; Rothe, 2024). In fact, the “sensory work” described in this collective discussion piece can “never be wholly distinguished from the material practices and modes of technological interface and analysis … via which they are produced” (Johns, 2017: 61). In the same vein, Suchman in the introduction to Klimburg-Witjes et al. (2021) calls for a focus on the socio-technical infrastructures of sensing “deployed in the name of securitisation.” Their edited volume shows the plethora of sensing technologies deployed for the purpose of security, surveillance and social sorting and, thereby, it also broadens the scope of security studies, where human language and imagery have long been focal points of analysis (Bleiker, 2018; Buzan et al., 1998; Callahan, 2020). A prime point of departure for our joint discussion is, therefore, to orient the analysis towards “sensing technologies” (Gabrys and Pritchard, 2018). Sensing technologies are not mere intermediaries, giving practitioners direct access to a world that needs to be secured. Sensing is a technologically mediated and relational practice, in which the experience of security becomes distributed among human and nonhuman actors.

Three Contributions

Drawing on this understanding of “sensing security,” our contributions in this collective discussion piece focus on three interrelated aspects. First, we seek to draw attention to how sensing technologies are often believed to imitate and extend human senses, despite the wider range of affordances offered by such technologies. For instance, surveillance cameras and dialect recognition systems are often designed to mimic human senses such as seeing and hearing, closely relating it to what is assumed to be a human way of making sense of the world. At the same time, despite being built with human senses in mind, sensing technologies introduce novel affordances. Many emerging sensing technologies go beyond what human senses can do. Technologies such as sonar, radar, infrared cameras, or (unsupervised) machine learning algorithms register signals that are undetectable by the human sensorium (Fish, 2022; Garrett and McCosker, 2017). The unaided and unenhanced human eye is ill-equipped to see “the forest for the trees” or to identify “the needle in the haystack” in large and complex data sets (Aradau, 2015). Sensor technologies often generate an excessive volume of data that is difficult for humans to analyze without leveraging data analysis tools and software. Therefore, despite the ideal to imitate and extend the human sensorium, the affordances of sensing technologies are qualitatively different from those of human senses.

The second aspect that is highlighted in our individual contributions on “sensing security” is how sensing technologies amplify specific aspects of reality while reducing other aspects of the world that need to be secured (Ihde, 1990). What these sensors detect as a security threat—and what not—is entirely dependent on the specific type of sensor solution that is applied to address the security problem at hand (Klimburg-Witjes et al., 2021: 24). In fact, each of the sensory technologies we discuss in our individual contributions prioritizes certain information (visual, tactile, and auditory) whilst ignoring others. Camera sensors, for instance, capture movements and transform these into machine-readable data. These inputs can subsequently be transformed into sensorial outputs (discursive, visual, and numerical) that security professionals can interact with. The point here is not that “real life” cannot be translated into technological format; it is that this translation can be done in multiple ways, and often requires multiple processing steps, which include and exclude certain aspects, with variable political effects (Scheel, 2019). Sensing, thereby, is productive of multiple worlds to be secured (Gabrys, 2016; Mol, 2002). In other words, our collective discussion underscores that the ways in which security is made sensible and perceptible is never a neutral process.

This brings us to the last aspect that connects the individual contributions, which is that sensing technologies also perform distinctive forms of power. That is, the power to discriminate against persons; to transform the way they are acted upon; shape the character of borders and warfare; and “(re)draw boundaries and lines of exclusion” (Amicelle et al., 2015; Rothe, 2017). Isin and Ruppert’s (2020) notion of “sensory power” captures this dynamic, whereby data practices chart a new and emergent form of power (see also Rouvroy, 2011; Amoore, 2020). With the increasing number of sensors that can produce data, and the infrastructures that can process this with new techniques of aggregation and correlation, our collective discussion piece underscores the argument that the “very experience of characterizing the world’s conditions, and of exercising power to govern, make legally significant decisions, … are under revision” (Johns, 2017: 60).

While our individual contributions do not go as far to make paradigmatic claims about shifting power relations, the collection attests to the fact that sensors that produce data for these algorithmic forms of government are emerging in many security-related contexts: ranging from battlefield, borders, to our very own universities. We build on the argument that algorithmic forms of security pose a serious challenge, for that they “necessarily involve new ways of reading signals, detecting disturbances and shaping action” (Amoore and Raley, 2017: 5). Indeed, the era of computers as passive data repositories and processors is behind us. Machine learning algorithms can interpret a variety of data and are increasingly able to act autonomously, “operat[ing] both beneath and beyond the threshold of human perception” (Amoore, 2020: 30; Aradau and Blanke, 2022: 172). However, we underscore in this collective discussion that sensing security does not have to be digital or automated; paper-based or “low-tech” forms of mediated sensing remain relevant in security settings (Bonelli and Ragazzi, 2014). Furthermore, we do not want to exaggerate the impact of the algorithmic sensing technologies in the fields we discuss by acknowledging their situated and contingent character (see also Aradau, 2023).

Our Collective Discussion Piece

To sum up, the goal of our collective discussion is to draw out some of these specificities of sensing security, which consist not only in showing how these sensing technologies partake in a re-ordering of perception, but also in exploring how they are productive of new practices of decision-making and securitization. The idea for this group discussion came from a workshop led by Klaudia Klonowska and Sofie van der Maarel in September 2023 at the European International Studies Association (EISA) conference. Afterwards, Jasper van der Kist, Natalie Welfens, and Ruben van de Ven joined. Through regular meetings and discussions about our respective contributions over a time span of more than one year, we developed the shared understanding of sensing security presented here, grounded in our diverse disciplinary backgrounds and empirical engagements. What unites us is a common concern with the non-neutral ways security is made perceptible and actionable through various forms of mediation. The concept of “sensing security” has proven to be fruitful in our collaborative discussion and individual inquiries—not as a rigid framework but as an invitation to focus on the ways in which perception, action and power intersect. This collective discussion is intended to push the debate on “sensing security” and to stimulate further thinking about the relation between sensing and securitization, as well as more research on mediated practices and technologies of sensing in security contexts.

The piece proceeds as follows. First, Van der Maarel discusses the military’s development of robotic technologies, emphasizing the focus on replicating human sensory capabilities rather than surpassing them. Second, Van de Ven examines the operations of “transduction” that allow sensor data of classroom occupancy to be computed and made actionable for the purpose of epidemic control in higher education. Third, Klonowska discusses “single pane of glass” as a primary interface for the military that allows for increasing amounts of heterogeneous sensors and data points to be algorithmically processed and visualized in a central location to identify unseen patterns and provide an operational view of the battlefield. Fourth, Welfens reorients the collective discussion towards embodied experiences of security, situating perceptions and judgments in refugee admission programs within the complex interplay of frontline workers and their mundane technological tools. Fifth, Van der Kist discusses the mediation of a particular sensing technology—speech biometry—in refugee status determination processes, emphasizing the discrete manner by which linguistic identities are made perceptible to decision-makers and how this has direct implications for how asylum seekers can present themselves discursively as a political-legal subject.

Sofie van der Maarel

As an anthropologist studying security innovation through the framework of STS and CSS, I am interested in lived experiences of security practitioners working with new technologies. This section zooms in on military experiences of envisioning, developing, and testing sensor technologies. Specifically, it discusses how these technologies are envisioned as an imitation and extension of human senses to achieve an ideal of gathering and analyzing as much data as possible. Focusing on illustrations from a military case study, this contribution explores sensing in relation to military practices and perceptions of threat and enemy targeting.

Central in the desire of militaries to implement such technologies is its assumed potential to increase “situational awareness,” understood as the reading and making manageable of uncertain situations (Hentschel et al., 2025). Approaching situational awareness as a “governmental and everyday practice of decision-making and manoeuvring in highly uncertain settings” (Hentschel et al., 2025: 1), it also impacts the lived experiences of those working with sensing technologies. Reframing threat perception and response, these technologies shape assumptions of normalcy and security. As also discussed in the introduction, sensing is thus not a neutral or objective assessment of threat and risk, but a situated and embodied practice. This contribution further highlights the situated ideals of replicating human sensory capabilities.

In relation to the other contributions, sensing in this section revolves around gathering data before it is processed for decision-making practices. It draws on illustrations from a case study with the Dutch Army’s Robots and Autonomous Systems Unit. Soldiers in this unit experimented with semi-autonomous vehicles produced by Estonian company Milrem Robotics. This company focuses on “the synchronized deployment of soldiers, manned and unmanned air and ground vehicles, robotics, and sensors to achieve enhanced situational awareness, increased lethality, and improved survivability” (Milrem Robotics, 2023). According to the manufacturer, the vehicle is built for tactical reconnaissance missions and includes a “variety of sensors for day and night operations, acoustic gunshot detector, smoke screen protection and a ground surveillance radar [which] allows units to do multi sensor identification” (Milrem Robotics, 2023). The vehicle additionally contains two sensors to detect weather conditions: the wind direction and temperature.

Imitating Human Senses

At this Army unit, soldiers and manufacturers closely collaborated at a military base in the Netherlands to make the vehicle, which was earlier used in forestry, fit for military purposes. These actors decided that the vehicle must have two cameras that together would create 3D images. This idea was based on the nature of human eyes, in which the registrations of two eyes are made into one image in the brain. However, the vehicle does not have a human brain and the software behind it struggled to integrate two images into one coherent livestream. For people working with the vehicle, this was very confusing, and a lot of extra work was required to monitor, process, and integrate images from two screens. Such technological developments with the vehicle’s sensors followed the rationale of human senses. Whereas reasoning from the technological functioning, manufacturers mentioned that it made much more sense to have one fisheye lens camera, one “super eye.”

The extension of human senses through technologies was also discussed in relation to military maneuvers with the vehicle. As one soldier described:

Human senses have to be transferred to such a device. As humans we are pretty well developed, we register a lot. So far it has been difficult to get all our human senses in a device. For example, the vehicle was driving without a microphone, it was basically deaf. And it was being shot at from the flank and no one noticed it … If you are a human being walking there, you notice it if suddenly there is a sound coming from the side. With the robot … it does not see things like that … so one of the points we raised was to integrate acoustic sensors … seeing and hearing are the most important senses in technologies.

After that incident, the vehicle was equipped with four acoustic sensors that could pick up sounds and estimate what kind of weapon was used by potential enemies and at what distance. These sensors were not yet used in practice, but it remained soldiers’ wish to implement them in their operations to detect as well as to produce sounds. In that way, according to these soldiers, they can also be used to disturb the enemy and to be able to drive the vehicle towards enemies to pass messages, such as “surrender yourself.” The soldier explaining this admitted that this had “high killer robot” characteristics.

One military operator, moreover, stressed that the systems should add value to their operations in the form of extra eyes and ears, but “you shouldn’t be too busy with the thing.” If it becomes too complicated to operate, he stated, you can no longer look around and “you lose your own perception, because with your own eyes, you best see the enemy coming.” So, the sensors were only seen as valuable if they extended human senses, but were considered a burden when they distracted or diminished those. Operators thus ultimately wanted to maintain a sense of control by using their own senses to gain situational awareness.

These illustrations, in particular the fisheye camera and sound sensor examples, show how soldiers and manufacturers idealized sensing technologies as an extension of human senses, for example, more ears and eyes on the ground. Not the capabilities of the technologies were taken as a starting point, rather the focus was on how these technologies could mimic and extend humans’ physiological ears and eyes. Furthermore, many technologies were developed with the promise of distancing soldiers from the battlefield, offering the potential for remote operations that would enhance their safety (Van Der Maarel et al., 2025). However, the emphasis on the human senses in sensing technologies reveals a desire to keep the human involved, to gather data “as if” they were fully present. Almost as though soldiers were sensing with their own senses, only processing higher quantities of data. Realizing that this is unattainable as the large amounts of data of the “most complete image possible” cannot be analyzed by humans only, considering their cognitive limits, the focus shifted towards automation.

“Most Complete Image Possible”

While the focus in the unit was on sensing and the use of sensors in data gathering, or in military language, “tactical reconnaissance” or “situational awareness,” soldiers and manufacturers also discussed the analysis of this data. In doing so, they highlighted the importance of combining sensors. One soldier stated: “why do human eyes need to see it? Why don’t you automate that so that with an optical sensor you have an image, and that image is immediately recognized by the software.” He used an example of Russian troops:

Look, what you ultimately want is to generate the most complete image possible with such a [combination] sensor box. Now a soldier is sitting somewhere, and he observes, and hears or sees something and then the message is: “there are a thousand Russians coming towards me.” But the soldier 50 meters away says: “there are only 50.” So, what we want is that the data generated by the sensor box is as objective as possible and that it is therefore much more useful in the rest of the process. So, then you want to combine data, from the acoustic sensor, from the optical sensor, all kinds of things, and that all has to be integrated into a computer program. Well, where we’re at now is that all that data appears separately on a separate screen, and that is then analysed by some people who are knowledgeable, and they can make a complete picture of that.

These visions can be seen as ideals of sensing for security containing expectations, hopes, and desires to generate the “most complete image possible,” which is considered similar to the “most objective” image (this also links to Klonowska’s contribution on making security visible). Good sensing is thereby equated with more technologies, more data processing, and more integrated sensors. A combination of sensors and an integrated, automated analysis is considered more reliable than human judgment. Ironically, as mentioned earlier, human beings are the masters of “integrating senses” as our physical bodies are “programmed” to combine what we see, hear, smell, or touch to decide our course of action. So here again, the sensing technologies follow the logic of human sensing. According to the soldier mentioned above, the judgment would “normally” be based on two soldiers’ observations. The emphasis on combined sensors and automated data analysis suggests that this accounts for more accurate and “objective” sensing than those two soldiers (and all their senses) combined. These ideal visions of sensing closely relate to Haraway’s (Haraway, 1988a) discussions on the “god’s eye trick,” namely the desire and attempt to be able to see everything from nowhere. In this instance, it refers to the gathering and interpreting of information, whereas Welfens later discusses its relations to decision-making and positionality.

Moreover, focusing on how soldiers envision and design sensing technologies for security shows how these technologies and human-machine interactions are approached as an instrumental way to reach an enclosed world. This idea relates to the rationale that Suchman (2023) describes as ‘a resilient fantasy of data-driven, comprehensive command and control.’ Sensors, signal processing, and data transmission create an excess of data that threatens to destabilize the “technopolitical imaginary of just-in-time information, [and] AI is advanced as the promissory solution to automating data analysis and reclosing the world” (Suchman, 2023: 762). Soldiers’ discourses and ideal visions of sensing also evolve around gathering as much data as possible to gain situational awareness, or real-time understanding of what happens on the ground. Above that, a lot of trust is placed in automation processes to enhance soldiers’ sensory capabilities. Instead of altering how they see, hear, or sense, which is possible with novel technologies, these processes aim to augment their natural senses by gathering and analyzing more data rather than different types of data. From this perspective, sensor technologies are thus approached as imitations and extensions of human senses, and sensing security links to ideals of achieving security through generating the most complete sensing possible.

Ideals of Military Sensing

Concluding, this contribution explored sensing in relation to military practices and experiences of envisioning and testing robotic technologies. These were based on situated discourses and ideals, embodying military perceptions of what is considered a security threat. Compared to the other contributions, this section emphasized the human ideals of sensing technologies, showing how these humans approached them during their development in an instrumental and functional way. In sum, this contribution makes two observations about how security practitioners envision, design, and develop sensing technologies.

First, the human-nonhuman interactions in this illustration concern the idealization that robotic technologies should “sense” in the same way as humans sense. Following this logic, attention to sensing security in the military’s development of robotic technologies shows an emphasis on replicating human sensory capabilities rather than surpassing them. Second, this lens reveals the prevailing idea that the more senses are combined, the more objective the understanding of situational awareness. Hence, security knowledge is perceived as objective when mediated through combined and integrated sensing technologies.

These observations, however, reveal a paradox. As became visible, the more data is integrated, the more there is a need to use AI-enabled systems to analyze the large amounts of data. So, whereas the emphasis in envisioning and developing military sensors is on imitating and replicating human senses, this cannot be maintained for the analysis of the raw data. To turn this data into actionable information, military units such as the one discussed in this contribution rely increasingly on data analysis tools and software. As such, even though sensing technologies are often understood as imitating human sensorium, their purpose is to move beyond the human perception and register signals or patterns that are undetectable by the human sensorium (Aradau, 2015; Fish, 2022). Therefore, sensory affordances may be different from human sensing, which came to expression in the example of the fisheye camera. Envisioning sensors as extended human senses, thus, has its limitations for sensing security.

Ruben van de Ven

… if our ears were ten times more sensitive, we would hear matter roar– and presumably nothing else (Kittler, 2018: 352–353).

In her contribution, Van der Maarel considered ideals of sensing security for military data-gathering practices. In my contribution, I want to dig a bit deeper into this relation between data-gathering and sensing. For what are the politics at play when data-capture and processing become a sensing practice? In other words, what is at stake when we call something a sensor? By examining the constitutional moment of the sensor, I bring two things to the discussion. First, I want to engage in a relational account of sensing security that moves beyond the primacy of human-machine relations but also engages in the tensions that appear between various stages of machine processing. This will help me to upset the dominant narrative that sensors would provide “the most complete image possible,” as Van der Maarel’s respondent suggests, and instead tease out the primary function of sensors: a transduction of the environment, which is as much a reduction as an amplification of reality (see Mackenzie, 2002; Simondon, 2011). Like rose-colored glasses, sensors filter out much detail (which is considered noise), while painting everything in their color. Thus, examining the construction of a signal by tracing its transductions has a consequence for the dominant idea of the dashboard as providing an all-encompassing overview. This resonates with what Latour and Hermant (1998: 32) have called oligopticons, in which only “very little” is visible, but this narrow vision comes with “great precision.” The oligopticon is thus rather different from Bentham’s panopticon in which everything would be visible to the operator. As such, Latour proposes that both the megalomania of the operator and the paranoia of the watched are obsolete (lazy even!) responses to sensing practices. However, by examining the institution of such an oligopticon, I want to reconsider this implied depoliticization of sensing practices. Instead, I locate the sensor’s politics precisely in the negotiation of “very little” and “great precision” it enables; in the question of what is produced, what is filtered out, what remains in the processing of signals.

For this exploration, I turn to a case that happened at my own university in Leiden, the Netherlands. In November 2021, the student newspaper of the university, Mare, revealed that over the preceding months, the university had equipped many of its classrooms with devices intended to measure room occupancy in light of COVID-19 restrictions. Over 370 such devices—the PC2S developed by Xovishad been installed (Kloosterman and Reid, 2021). The university consistently called the devices “classroom scanners” and “occupancy sensors.” These devices, so Mare revealed in their publication, were in fact small cameras connected to a computer. They were able to quantify much more than room occupancy: built in algorithms to assess gender, race, and emotion of those passing by. After the publication by Mare, both students and staff rose up against these devices. Opponents critiqued the devices, calling them out for being surveillance cameras that pose a disproportional infringement of the privacy of any user of the buildings. The university board, in turn, apologized, not for installing the devices, but for their limited communication, implying the critique of these devices was due to a misunderstanding. After all, the devices should not have been considered as part of a surveillance apparatus, but merely as “sensors” that aid building management. Opponents, in turn, postulated that painting the cameras + computers as sensors was a rhetorical trick that downplays the surveillance potential of these devices.

What if, however, we take the framing of camera + computer as a sensor herein seriously rather than nefarious? This is not to simply sidestep the critique of these “sensors” as constituting a surveillance infrastructure. On the contrary, I want to use this case as a means to explore the ontological politics that are at play when an ensemble of technologies becomes designated as a single device. This is all the more relevant as this case of referring to camera + computer as “sensor” does not stand on itself. In fact, various manufacturers and professionals in algorithmic surveillance do so⁵. The critique of the camera + computer = sensor suggests that we can pry open these sensors to further examine these security devices (Amicelle et al., 2015). Which relations emerge if we draw out an exploded-view of the occupancy sensor?

To understand how camera + computer makes a sensor, let us briefly unpack the technological definition of sensors. One way to describe a sensor is to consider it as a transducer: a device that receives signals from a physical system, and which converts those into other signals that contain information about that system. For example, a temperature sensor changes its electronic conductivity depending on the surrounding heat. The vibrating movement of particles in the air is transduced into a variable electric current. The technological feat of this transduction of one signal into the other (the mundaneness of which only speaks to its marvel), thus, involves an epistemological step of erasure: those interested in the temperature signal are generally no longer concerned with the individual particle movements that help produce the signal. Of concern is that which emerges out of quantifying the kinetic energy of many particles: the temperature of a substance is an emergent property. Likewise, discussing the water control systems in Paris, Latour and Hermant write: “Their wisdom is proportional to their deliberate blindness. They gain in coordination capacities only because they agree to lose first water and then most of the information” (1998: 28). A “catch-all” of data collection largely contains noise, which needs to be discarded if one wants to be able to act on the data. This seemingly resonates with the claim by the board of Leiden University that the camera + computer = sensors do not surveil individuals but “measure” the occupancy of rooms. Much like in the example of the temperature sensor, room occupancy is merely considered as an emergent property that follows from the movement of people in and out of a space. However, following the work of Simondon, media scholar Mackenzie (2002) argues that an attention to signal transductions “aids in tracking processes that come into being at the intersection of diverse realities. These diverse realities include corporeal, geographical, economic, conceptual, biopolitical, geopolitical and affective dimensions” (2002: 18). Transduction is not simply reduction: at each step, the sensor filters, mobilizing assumptions about the use of the data in order to throw away most of it, recasting what remains in its own light. An attention to transductions helps to outline the articulations of security that are at play in the production of sensor signals.

Producing an occupancy “measurement” involves a huge number of computer instructions, which I here bracket to five consecutive operations. First, light hits a camera sensor, creating an electrical current, which can be represented as a stream of bits to be computationally interpreted: the signal produced by the image sensor. One should not underestimate the quantity of data produced by an image sensor: even a rather standard HD frame of 1920 by 1080 pixels adds up to more than two million pixels. Assuming three colors, and eight bits per color, this adds up to 6.22 megabytes per frame. At 30 frames per second, such a device might produce over 180 megabytes per second. Collectively, 370 of such image sensors produce over 66 gigabytes of data every second. The electronics processing these signals simply would not be able to keep up, were it not for the fact that image and video encoders throw away vast quantities of the information. To do so, in a second step, the JPG compressor (used in the common Motion-JPG video format) transduces images into a series of wave functions, in a grid of 8 by 8 pixels. This removes fine details from the image, while keeping the overall structure perceptibly similar to the input image—“perceptually” here denoting the limitations of the human eye (another, often used pixel format is YUV 4:2:0, which also employs a reduction of pixels based on assumptions about the sensitivity of human vision). In a third step, the camera + computer = sensor takes this algorithmically processed image signal and predicts where people occur in the image using computer vision techniques such as convolutional neural networks. This creates another signal, one of distinct detections. In a fourth step, the detections in one video frame are compared with those of the previous frame using an object tracker. In the fifth and final step, a fluctuating occupancy signal can be produced by performing a rather simple integration of the tracks into a numeric state: in and out of the room, +1 and -1. What is left is the count, a numeric value, fluctuating in time: room occupancy as a signal⁶.

Despite each transduction further obscuring the relation between the individual subject and those operating the system (Aradau, 2023: 237), it is not eradicated. On the contrary, as much as each transduction erases, it also articulates this relation. Video encoders produce images for human consumption, while the images are processed by algorithms, and detections in individual frames are glued together based on assumptions of how humans generally move. Each step of the process produces an additional rendition of the individual. While all visitors to the buildings were being filmed and counted by the camera + computer = sensors, not all were equally governed by this control system. The cascading transductions displace the device’s primary subject of governance. For instance, on some occasions, teaching staff got warning SMS messages indicating their room had too many occupants, and they were likely violating COVID-19 restrictions. University staff used to make their own judgment about the number of students they let into their classroom; now, the agency of intervention has moved to an automated system that signals the staff about occupancy violations. This was not received well by many staff members, who considered these interventions as an expression of sheer lack of respect from the university board towards its staff. So, while the device’s object detector is oblivious to classroom hierarchies—scanning students and staff alike—(Lianos, 2003), the sensory assemblage imposes new hierarchies between the operator and the subject.

In the case of the people counter, by placing “real time” sensors in classrooms, a logic of “smartness” enters the university building. As Halpern and Mitchell (2022) put it: smartness is an epistemology, a “knowing and representing the world so that one can act in and upon that world” (2022 xi). It was the continuous capture of sensor data that greatly increased the temporal resolution of the data available. With minute-precision measurements of occupancy, it became possible to not only optimize and anticipate, but also to intervene. The transduced signals form what Isin and Ruppert (2020) call “live clusters” that act as “intermediary objects of government between bodies and populations” through which such sensory assemblages facilitate real-time control. Sensing security is not just about producing highly specific measurements: constant, real-time monitoring often becomes convolved with (automated) intervention.

The relation between data-gathering and intervention lies not only in the immediacy with which it happens, but also in the precision that the sensor’s cascade of transductions affords. To make accurate control with the occupancy sensor possible, each individual needs to be detected, marked, and added to the occupancy count. To exemplify this, the story of Leiden University can be contrasted with another occupancy counter, one used by the primary Dutch railway operator. In their app and on their website, Dutch Railways indicates the occupancy of trains. They are able to provide this information without counting individuals entering the carriages. Already for a long time, trains have been weighed on particular parts of the railway track. The Dutch Railways use these measurements to estimate the number of people in a train. Such estimates suffice to monitor train usage but are not precise enough for direct intervention over the individuals on board. The university’s occupancy counters had been introduced to curb the COVID-19 crisis, thus their resulting signal—room occupancy—can be considered as a proxy for a property more relevant in light of the epidemic, namely the room’s air quality. Had the air quality been measured directly (such sensors do exist), an estimate of occupancy would have been possible. Nevertheless, such an estimate would not have been precise enough for direct intervention towards the teaching staff. In short, the accuracy with which signals are produced is of relevance to the governing practices afforded by sensing security.

I have discussed sensing security as an attention to the transductions that are happening in the various stages of signal production. While “sensors” are generally conceived as mere input points of data, they are devices that take on computational work (Pelizza and van Rossum, 2021). This attention to the sensing device counters the depoliticization of mere “measurement instruments” in two ways. First, much like Latour’s take on the oligopticon, tracing transductions renders the assemblage as an exploded view, allowing us to inspect the relations that are drawn into the reduction and amplification of reality. This is what we saw in the initial critique of the people counter: calling the device a sensor brushes over the complex sequence of operations that takes place. Thus, opening up the occupation sensor forces acknowledgment of its composition as camera + computer; the sensor is a device that produces a video stream. The 370 sensors are 370 cameras that become latently part of an already existing infrastructure for surveillance. Thus, the composition of the sensor device matters, for a politics of the sensor does not replace that of its components, it multiplies them. Second, tracing the cascading transductions of the camera + computer = sensor locates the politics of its aggregated signal in the negotiation of immediacy and precision as facilitators of intervention. After the uproar, the university board decided to take down the camera + computer = sensors. To still get a grasp on room occupancy, the faculty are back to sending around people with a notebook. At set intervals, they briefly interrupt presentations and conversations by opening the door and counting the people in the room. Like the “occupancy sensors,” they produce a numeric outcome. But, relying on their human forgetfulness, they are seen by many as less of a privacy hurdle. They rely on sight, but not on the production of image artifacts. Moreover, working in silence, the human counters abide by traditional classroom hierarchies and are perceived as less threatening to the teaching staff’s authority. The real-timeness of the measurements has decreased, but usage of classrooms can still be optimized and anticipated, while staff have regained the agency for intervention.

Klaudia Klonowska

Other interventions in this collective discussion attend to the diversity of sensory inputs and the computational operations that transform sensory inputs, such as sound bites or electromagnetic waves, into readable signals. I suggest that the study of sensing security additionally entails attention to interfaces, or in other words, ways in which computerized sensory inputs are made visible and received by human users. Interfaces bring attention to how visibility becomes simulated and “titled phenomena appear coherent while obscuring the layered, socio-technical operations that produce and reproduce them” (Johns, 2017: 10–11). In this short intervention, I am primarily interested in how aspirations for greater situational awareness are operationalized, what picture is attained, and how it is perceived and made actionable.

I come into this collective discussion with a background in international humanitarian law and an STS-inspired interest in what security assemblages enable in terms of legally relevant military decisions and conduct. Though “sensing” is often overlooked in legal research, several authors have underlined its relevance to matters of justice and governance. Most notably, Johns (2017: 60) has argued that jurisdiction gets exercised only when the legally authorized agent “gain[s] both the capacity and inclination to sense a certain condition, person, or pattern.” Hamilton et al. (2017: 2) further argued that vice versa, available “[w]ays of sensing shape ways of knowing and the regulatory projects upon which they rely.” As the intervention of Van de Ven shows, attention to the socio-technical practices and complex transformations that take place when sensory technologies are employed provides opportunities for critical engagement with the implications to the exercise of human rights. In these spaces, lawful interventions of security actors rely on the capability to visualize threats. As this contribution will further argue, paying attention to the interfaces of sensing technologies and their users can tell us more about the distribution of authority and power to intervene.

In this piece, I draw insights from interviews with practitioners of the United States Task Force 59 (TF59) conducted between March and May 2023. TF59 is a naval unit with two operational hubs in Manama, Bahrain and Aqaba, Jordan that experiments with and deploys cameras, sonars, thermal radars, aerial and underwater drones, unmanned vessels, and algorithmic tools to sense the waters of the Red Sea, Gulf of Oman, Persian Gulf, Arabian Sea, and parts of the Indian Ocean (Brandi 2023). The main purpose of this extensive endeavor—which includes one of the world’s biggest fleets of unmanned vessels—is to detect and identify threats, and in turn improve maritime domain awareness. TF59 serves as an excellent illustration of a broader trend of how the selection of “relevant” data and the performative associating of data into profiles contribute to the coherence and operability of the common operational picture.

“Single Pane of Glass”

One of the major paradoxes of the digital era is that the volume of data that we produce and collect is so large that it is neither manageable nor digestible by humans alone, which also became clear in Van der Maarel’s case. While data-gathering techniques improve, new solutions such as machine-learning algorithms are needed to manage its volume and represent data in an accessible format. Vice versa, as data-hungry machine-learning systems are developed, the need for data-gathering grows in an endless cycle. For many security actors, a solution to the problem of data overload is the design of databases and visual interfaces that serve as access points. At TF59, this solution is called a “single pane of glass.”

A single pane of glass is based on the idea of a visual representation of an operational environment in which data from all kinds of sensors and data-gathering devices mounted on unmanned vessels are pulled together to create a uniform and coherent picture of what is happening “out there” on the open seas. This picture is based on signals from a wide variety of sensing devices, including satellites, drone cameras, and multibeam sonars. The term “single pane of glass” is meant to highlight that—instead of looking at the world through several different computers and “windows”—this technology enables the military to rely on one interface, one “window” through which visualization of the operational environment occurs. In this process of data centralization and visualization, the aspiration is to improve the common operational picture of security actors, meaning their shared awareness and understanding of emerging threats in the maritime domain.

The choice of the single pane of glass entails that the multi-model sensorial data that is gathered from various devices is reduced to a uniquely visual form of representation that relies on pictures, maps, colorful graphs, indicators, and heat maps. Sometimes access to sound may be retained, but likely the single pane of glass prioritizes the visual at the expense of other senses that are made remote. Though it should be of note that human operators can follow up on the recommendations presented by the single pane of glass and deploy a patrol boat to waters to gather “more information” using their physiological senses. The single pane of glass is not an accidental form of representation of senses; rather, it shows that the well-reported phenomena of “ocularcentrism” still live on, described as a tendency of humans to prioritize the visual experience in making evidentiary representations of the world (Hamilton et al., 2017: 4). These evidentiary representations, which visualize risks on the waters, then mediate the sensing experiences of human operators and inform their decision-making processes.

Constructing a Coherent Picture

The creation of the single pane of glass, however, is not a straightforward process of representation of what is “out there.” Instead, it is a complex process that depends on both human expertise and choices, as well as algorithmic processing and visualization practices that construct what appears to be a complete picture. In this process, selection of sensing devices, collection of data, and choices about risk profiles contribute to the sense of coherence of the single pane of glass.

The main challenge for the creation of the single pane of glass is the choice of which data is “relevant” so as to be visualized and presented to the human operators. Machine-learning algorithms are critical to this process. Amoore (2020: 9) remarks: “the very essence of algorithms is that they afford greater degrees of recognition and value to some features of a scene than they do to others.” Algorithmic filtering is thus a process of selection whereby certain data is rendered “relevant” and visualized. As opposed to filtering, which relies on the removal of unwanted information, selection highlights the process of adding together. What “relevant” in these contexts means is an outcome of socio-technical chains of interactions, influenced by the trade-offs of engineers in the algorithmic model design, abnormalities in the emerging patterns compared to “normality” observed in human pre-selected training data (see also Pasquinelli, 2019). Ultimately, social ideas about what a threat “looks like” in these security contexts will influence the design and iteratively selection of “relevant” features.

In the case of TF59, the selection process is aimed at automating the detection of maritime vessels exhibiting “abnormal” behavior, indicating potential smuggling of weapons or drugs, piracy activities, human trafficking, or illegal fishing. To identify and visualize “abnormal” behaviors onto a single pane of glass, the system swifts through data points from the Automatic Identification System (AIS), which is a database that has been used to monitor maritime traffic since the early 2000s. AIS data is a numerical geolocation output produced by the collection of satellite imagery, radio signals, representation of location on maps, and human input. The use of AIS for the analysis of maritime traffic has a long history, but the use of machine-learning algorithmic techniques to automate the processes of sensing and knowing which vessels to look further into is relatively novel. Through this automation, the promise is that more vessels with abnormal and suspicious activities are detected so as to enable lawful interventions. In this process, the algorithm creates added layers of complexity to identify “abnormal” patterns: it uses both human-selected “features” (such as the behavior of two vessels meeting at sea which for long has been considered to indicate risky behavior such as smuggling of materials), as well as self-generated “features” that the algorithm develops while “learning” from training data and subsequent use.

Counterintuitively, the creation of the single pane of glass does not only rely on existing data, but in order to provide a sense of coherence it also relies on the predictive capabilities of machine-learning systems. As algorithms selectively visualize relevant data, they also simultaneously constructively fill in the gaps. The creation of a seemingly coherent picture that informs how vessels are to be classified into risky/non-risky or normal/abnormal maritime behaviors relies on the associative power of machine-learning algorithms to produce profiles.

For example, data from satellite imagery is collected in up-to-15-minute intervals, which results in temporary invisibilities of the movements of vessels on the seas. Machine-learning algorithms are used to fill in those invisibilities with visibilities, in other words, to provide a visual representation of a possible direction of movement of those vessels in the periods where sensory signals are missing. This is done by comparing common patterns of movement (in specific routes, for a specific type of vessel, in specific regions) and predicting the possible pathways that the vessel might take. In this way, despite missing data, the vessel’s movement is estimated and used to determine its classification as normal/abnormal. By filling in the gaps in the movement of vessels, the machine-learning algorithm contributes to the drawing of trajectories onto maps and graphs so that objects can be visualized and sensed by human operators.

The result is a seemingly coherent and operationalizable picture that can be used in a decision-making process. The creation of a single pane of glass, however, is not an objective representation of reality; rather, it is a process through which new layers of visibility are produced. The co-production of visibilities is where “relevant” threats/risks/bodies are defined, and gaps and invisibilities are filled by the meticulous work of the algorithms, technical developers, and human users. In this process, the technology redistributes which vessels can be sensed by security actors, how they are made sense of, how many spaces can be sensed at the same time, and, therefore, how many bodies and movements can be governed and acted upon at the same time.

The Power of Differential Visibilities

The choice of creating a single pane of glass is meant to serve as TF59’s common interface for the detection of threats on the open waters. It gives the impression of centralization. Indeed, the technical pursuit of a single pane of glass means that data formats are standardized, and data is stored in shared clouds. The centralization of data through cloud computing also means that it can be easily accessed by different units, making the single pane of glass available to various security actors. The aspiration is that command posts, as well as different deployed naval units and other states or partner organizations with whom they are cooperating, can all have access to the same data infrastructure and act upon a shared (meaning standardized) understanding of the emerging threats. Naturally, this aspiration is not Navy-specific; similar projects of creating the single pane of glass are underway in the Army and Air Force. For example, the US Army is advancing its “deep-sensing” initiatives to improve the “ability to see and sense deeper, delivering an organic collection capability” (Baldwin, 2024; Klonowska and van der Maarel, 2025).

Sensing is often presented as an organic, natural, and even neutral act of information collection. This understanding, however, downplays the important connection between data accumulation and information processing capabilities on the one hand and the ability to “act on a distance” on the other (Miller and Rose, 1990; see also Van Der Kist et al., 2019). In his book Reassembling the Social, Latour (2005: 181) notes that the command post’s capacity to control stems from their ability to maintain such large networks of circulating references accounts for an essential form of power:

It is obvious, for instance, that an army’s command and control centre is not “bigger” and “wider” than the local front thousands of miles away where soldiers are risking their life, but it is clear nonetheless that such a war room can command and control anything—as the name indicates—only as long as it remains connected to the theatre of operation through a ceaseless transport of information.

In line with Latour’s argument, the informational-material arrangements that condition and make it possible to “see” and act upon the operational environment from one single command point, as long as the “connections” through which information travels “hold” (Latour, 2005: 181). From this point of view, my example of the single pane of glass illustrates how this sensing technology is intricately related to the accumulation of power.

Whereas the technological developments are pulling in the direction of greater diffusion of control centers, at the same time, the military institution, married to the ideas of hierarchization, implements new strategies to control the flows of data. One such strategy is the layering of the single pane of glass. This is also another means by which interfaces and sensor data are made operable. The single pane of glass is meant to be scalable, so that actors can be assigned different access depending on their clearance level. Such representations are then multi-layered so that a commander has access to more detailed information versus an operator whose access may be limited. A corporate partner may also be excluded from access to classified data and hence may have access to only a certain layer of the single pane of glass. This layering feature allows the military to manage which information is visible to actors that are connected to the system, and thereby also manage the control centers. Hence, attending to the interface and its various technical design choices brings attention not only to what information becomes sensible but also to how the capacity to act upon that information is arranged through the material-semiotic means.

To conclude, sensing security is not only a way to make objects/persons sensible but also a socio-technical means to distribute authority and power and make those objects/persons governable. Hence, attention to sensing security also entails attention to interfaces, which are the moments of encounters between computerized forms of sensed data and human users mediated by technologies, context, biases, language, and a variety of other subjectivities and materialities that make sensing and knowing situated. The turn to sensing also entails challenging technological determinism, a view that technology “as an inherent logic, insusceptible to external factors” by tracing human-nonhuman configurations and “reverse blackboxing” how things are rendered visible and invisible in those complex networks (Schiølin, 2023: 368), and how different interests are involved in the making of security threats. There are socio-politico-legal questions within each transformation. Scholars with a particular interest in legal considerations can attune to injustices that are produced through sensing regimes, when visibilities and invisibilities are stabilized (see Van der Kist’s contribution), when calls for transparency are mobilized, when resistance to visibility is expressed, and when access to sensing and hence knowing is unequal.

Natalie Welfens

In my research, I am interested in categorization and sorting practices in the context of refugee and migration governance and the social inequalities they produce. In this field, just like in other security contexts, the use of technologies is proliferating (Ozkul, 2023), triggering questions on, for instance, how technologies construe migration as a threat (Huysmans, 2000) and policy problem (Glouftsios and Scheel, 2020), or reinforce epistemic violence in asylum procedures (Scheel, 2024). Looking at other security fields, the contributions to this collective discussion so far have illustrated how “sensing security” can offer a fruitful analytical lens to render visible how technologies mimic and extend human senses, the politics of sensors as transducing devices, and the processes of visualizing sensory data to provide operational knowledge. These empirical examples have underscored the role that high-tech sensors and their algorithmic logics play in mediating security and rendering it actionable, but also the role that human interpretation of data and signals play.

In my contribution, I would like to deepen and complicate these conversations by focusing on low-tech tools and discretionary sense-making by human agents in the context of refugee governance. First, while high-tech sensors often dominate discussions of sensing security, my analysis highlights how also mundane inscriptions (forms) and technologies (phone calls) mediate categorizations of refugees as ‘at risk’ or ‘a risk’, as objects of care and/or of control (Aradau, 2004; Pallister-Wilkins, 2015). These administrative tools, though seemingly simple, are central to reducing the complex world of mobility and security into one-dimensional categories in order for it to be made sense-able and governable (see also Bonelli and Ragazzi, 2014). As such, they impact whose protection needs are heard, seen, registered, and addressed, and which ones are ignored. Second, my contribution sheds light on embodied practices and highlights the situated knowledge of frontline workers as a critical aspect of “sensing security.” This resonates with the observation that while technologies in migration governance proliferate and produce “control knowledge,” human case workers still often assume the positions of “epistemic gatekeepers” (Amelung et al., 2024; Scheel, 2024). Interrogating these aspects of human discretion in sensing security, I argue, is key for locating power in human-technology configurations (Suchman, 2006), and when examining the “redistribution of the sensible” (Rancière et al., 1999), as Van der Kist will expand on in the subsequent contribution.

The empirical context in which I will illustrate this argument is refugee resettlement. Through resettlement and similar refugee admission programs, Global North states admit limited quotas of the “most vulnerable” refugees from countries of first refuge, while excluding those they deem “risky” for the nation state. In selection practices, categorizations serve to assess, on the one hand, risks to refugees’ human security (e.g., exposure to violence, lack of access to healthcare), and on the other hand, the risks they may potentially pose to admission states’ security (Welfens and Bonjour, 2021). As resettlement is a discretionary, humanitarian commitment by states, spots are scarce: less than one percent of the world’s refugee population will be resettled. Therefore, categorizations of refugees’ risks and riskiness not only powerfully illustrate how classification systems stratify, include, and exclude (see also Bowker and Star, 1999), but also their real-life consequences. For that, as (Fassin, 2007: 500) notes, “the saving of individuals, […] presupposes not only risking others but also making a selection of which existences it is possible or legitimate to save.”

Building on Latour’s (1999) concept of the “chain of translation” and its development in IPS (De Goede, 2018), I conceptualize refugee admission as a chain of admission: a transnational process through which individuals in countries of refuge are transformed into legally recognized refugees via a series of interpretive and bureaucratic steps . At each link—spanning NGOs, international organizations, and state agencies—categories like “vulnerability” or “risk” are not simply applied but translated. As dossiers circulate, categorizations are reinterpreted through situated judgments and shifting institutional logics (Lipsky, 2010; Yanow, 2003). This translation is mediated through both low-tech tools (e.g., paper forms and spreadsheets) and more advanced technologies (e.g., digital case management systems, databases). Sensing security in this context then encompasses interpreting and translating this technologically mediated information by caseworkers based on their tacit or situated knowledge—i.e., “the very mundane, but still expert, understanding of and practical reasoning about local conditions derived from lived experience” (Yanow, 2003: 236).

These dynamics become tangible in the everyday practices of refugee resettlement, where sensing, categorization, and technological mediation converge in concrete bureaucratic processes. To illustrate, based on fieldwork in Turkey in 2019, I will focus on one particular link in the chain of admission: when referrals of people “in need of resettlement” reach the United Nations High Commissioner for Refugees (UNHCR), case workers make an initial assessment of people’s resettlement need and eligibility by means of a phone interview. Unlike the high-tech sensors in military operations and pandemic control described above, UNHCR, in its assessments of referrals, relies on more low-tech technologies, which, however, also mediate sensing in particular ways. For instance, UNHCR Turkey receives the list of initial referrals—Syrian refugees identified by the Turkish migration authority as having a “resettlement need”—in the form of a basic Excel spreadsheet. Next to a few columns with personal data, the reason for referral is reduced to one category, such as “medical needs,” “women in need,” or “survivor of violence or torture.” In this way, initial vulnerability assessments and the complex accounts of refugees’ situation in Syria and Turkey are condensed to a single line in an Excel spreadsheet, filtering out what is considered irrelevant “noise.” In that way, refugees’ lived experience is reduced to bureaucratic categories, stripped of its complexity and nuance, while amplifying what, from UNHCR’s and states’ perspective, enables the bureaucratic governance of mobile populations.

Such processes of reduction and amplification are not neutral and have significant real-life consequences. For instance, when resettlement of Syrians to Europe started, Turkish migration authorities had relatively little experience with assessing and classifying cases, which led to high numbers of referrals under the category of “medical needs,” as “these are the vulnerabilities that are often the most visible, the ones that the eye can see,” Mahmoud, the UNHCR case worker explained to me. Here, visibility, methods of measurement and their translation into technologically mediated categories are crucially political: what is deemed visible (e.g., physical injuries ) is what gets counted, while other forms of vulnerability (e.g., psychological trauma and social marginalization) are rendered invisible. At the same time, cases classified primarily as “medical need” can be particularly difficult to resettle, as many admission countries are reluctant to take substantial numbers of refugees with medical conditions. Even though these cases are clearly “at risk,” they are also considered “a risk” for admission states’ economic and welfare security.

UNHCR case workers like Mahmoud call the “resettlement candidates” submitted to him in an Excel spreadsheet one by one. The initial phone interview serves to translate the broad categories from the Excel sheet into UNHCR’s own digital case management file and to classify the cases along some of the criteria that UNHCR deems most relevant for further assessments: the protection needs people describe, whether they want to be resettled, where extended family lives and what documents they possess. At this link in the admission chain, through the digital case management system and its categorization logics, the various information gets reduced to ticking boxes. For instance, in the administrative form in UNHCR’s digital case management system the question, whether the family wants to resettle only allows for a yes/no answer—although refugees may not know what resettlement is, cannot choose the admission country and may first need to discuss the option with their family.

Initial phone interviews also serve to classify cases’ urgency. Mahmoud explains how his intuition and tacit knowledge, shaped by his Syrian background and lived experience, help him “sense” urgency in a caller’s voice. This practice extends human auditory perception, transforming spoken words—for example, a refugee’s description of their situation in Turkey and immediate needs—into actionable data such as classifying a case as “medium urgency.” This, in turn, indicates to the next case worker in the chain how swiftly they need to do a follow-up interview. This interplay between fixed categories and human discretion underscores the tension between technological standardization and human agency. While the Excel sheet and digital systems produce a durable, authoritative framework for categorization, Mahmoud, as a frontline worker, acts as an epistemic gatekeeper. His situated knowledge—his Syrian background, his fluency in Arabic, and his familiarity with the cities refugees fled from—enables him to interpret the “noise” that the system filters out. For example, when Mahmoud speaks to a Syrian woman on the phone and classifies her as “at risk” with “medium” urgency, he puts her on speaker so that I can listen to their conversation, though without understanding the content. It is in this interaction with reductive inscriptions and devices that Mahmood’s perception and cognition are amplified, allowing him to govern and decide whose cases count as urgent or not, which in turn significantly shapes refugees’ chances to be resettled.

Epistemic gatekeeping and the interplay of human agency and low-tech tools also matter in interpreting standardized rules for prioritizing and “deprioritising” cases. In UNHCR’s training manuals, there are some fixed categories, almost following an algorithm’s “when A then B” logic. For instance, at the time of my observations, when women were “without male protection,” they were automatically classified as “women at risk.” In contrast, someone who served in the military after 2011 in Syria, served for the Ba’ath party or had a high position in the government was automatically flagged as ‘a risk’ and deprioritized for resettlement. Similarly, cases of child marriage and polygamy are, despite their vulnerability, flagged as deprioritized in UNHCR’s digital case management system, making these people’s resettlement highly unlikely. Also here, the complexity of a case gets reduced to a bureaucratic marker, signaling a person’s risk and riskiness. Yet, the fact that translating refugees’ accounts into bureaucratic categories is human—not machine-made, allows for situated interpretation, weighing of factors and, importantly, exceptions to the rules. As Mahmoud explains to me, “For all deprioritisation cases there is an exception, because life matters.” For instance, when there is an urgent medical case, UNHCR still tries to process the case.

It is precisely such elements of discretionary judgments and sense-making that risk to get lost in algorithmic and, however rare, fully automated decision-making. In Germany’s federal admission programs for vulnerable people from Afghanistan, for instance, access to the program was largely determined by an algorithm-based scoring system. Instead of assessing individual applications based on detailed, case-specific justifications, eligibility was pre-filtered through a digital questionnaire with yes/no responses and box-ticking answers. Civil society actors filled in these questionnaires on applicants’ behalf, without always having a sense of how factors would be weighted and how the scoring system operated. Reviewers then only examined cases pre-selected by the algorithm, grounding their assessments on brief summaries, without provisions for nuanced evaluations and, notably, exceptions to security-related exclusion.

To conclude, as highlighted in the introduction to this collective discussion, sensing security as an analytical lens invites to interrogate the power dynamics embedded in sensing practices, including the question of how technologies redistribute power and authority. As the example of resettlement practices illustrated, this focus also matters for examining low-tech tools and the way they shape security perception and actions. Like high-tech sensors, they are crucially political in that they produce hierarchies of visibility, reduce abundance of data to one-dimensional categories and thereby “make up people” (Hacking 2006)—here, as promising “resettlement candidates” or deprioritized “security threats.” Yet, these low-tech systems, at least at some links of the admission chain, attribute substantial epistemic gatekeeping to humans and situated interpretation, compared to some black-boxed AI-driven systems. As Mahmoud’s practices illustrate, interpretation and situated judgments are informed by positionality, familiarity with local contexts, the ability to “read” a caller’s voice and, where necessary, overturn “when A then B” decisions. This embodied knowledge contrasts with the rigid categories of semi-automated and algorithmic systems that substantially reconfigure how power is distributed between human and technological actors—a theme Van der Kist will further explore.

Jasper van der Kist

In my research, I am interested in the entanglements of epistemic, technological and legal practices of refugee protection, particularly in relation to the procedures of refugee status determination. As noted by Welfens in her contribution, the fundamental principle underlying the operation of street-level bureaucracy is that it requires individuals to make decisions about other people (see also Lipsky, 2010: 161). While these discretionary practices have always been materially mediated in one way or another (Magalhães, 2016; Van Der Kist, 2024, 2025; Wissink, 2021), asylum procedures are becoming increasingly saturated by technologies affording new and divergent ways of sensing, perception and action (Scheel, 2024). In this contribution to the collective discussion piece, I will further underline the previous discussions on the mediated nature of sensing, in particular with regard to the generally overlooked auditory faculties (Ihde, 2007; Weitzel, 2018). Moreover, I would argue that a focus on sensing security has important political-legal ramifications, because it orders what can be said and perceived, and by whom (Rancière et al., 1999).

The State of Perception

Given the general tendency of officials to view the asylum seeker’s words with suspicion, and the perception that these words are unreliable and distorted, asylum administrations have long sought to outsource or develop in-house alternative forms of evidence (Lawrance and Ruffer, 2015). To assess the risk of persecution of the asylum claimant, for instance, medical techniques of sensing the body have long been used in asylum procedures to detect tangible signs of physical trauma (such as scar tissue) on the body of the migrant. Consequently, in this context, the concept of sensing security can be likened to the biblical narrative of Doubting Thomas, in which the sovereign is compelled to physically inspect the wounds of the stranger in order to validate their claims (Fassin 2012, 113).

In recent years, however, governments have introduced a plethora of computer-based and data-driven technologies ranging from physical and behavioral biometrics, automated document verification to the use of various predictive assessment and triage tools (see Ozkul, 2023 for an overview). This has mostly been motivated under the guise of creating a fairer, more efficient and effective asylum system; technological systems that can offer asylum administrations neutral solutions to a governmental problem, objectively perceiving the identity and risk of the migrant. Contrary to these views, however, I argue that the implementation of such technologies is never “innocent”—to invoke another one of Haraway (1988a) concepts. As technological mediators in their own right, these technologies introduce new and divergent affordances—that is, they enable and constrain alternate modes of sensing, perception and action (Gibson, 1979). More importantly, as I will demonstrate below, they may do so at the expense of other modes of experience.

Certainly, the contributors before me illustrate how sensing technologies are capable of perception and action. What has been implicit in the discussion, at least conceptually, is how sensing technologies relate to security decisions. The empirical examples given portray sensory prostheses to make strategic decisions on the “let live and make die” (see Van der Maarel); territorial “command-and-control” (see Klonowska); “classroom sensors” for university management to act on the unruly behavior of students and staff (see Van de Ven); or devices for administrators to determine whether risks should be attributed to the foreigner or the state (see Welfens). I would argue, therefore, that sensing security is intimately associated with sovereign practices of decision-making—or rather “any institution having the power to relate consequences to perceptions” (Stengers, 2018: 60).

What has remained implicit in the discussion, too, is how security perceptions are redistributed by the new “algorithmic” modes of sensing, and with what political consequences. This argument should not be confused with an all-encompassing “technological hype,” as we know from genealogical works, for instance, that the introduction of sensory inventions like the stethoscope gave rise to a distinct perception of sick bodies, changing the power dynamic in medical hospitals by subjecting patients to the authoritative gaze of the doctor (Rice, 2008). With the current invention of computational and data-driven technologies, we may be witnessing such a “redistribution of the sensible” (Rancière et al., 1999). What is important in a research agenda of sensing security is therefore not only what phenomena are made perceptible (and governable) by these new sensory technologies, but also remain sensitive to political questions of who (or what) remains imperceptible (see also Amoore, 2020; Aradau and Blanke, 2022).

I will limit my contribution here to a brief description of one of the technological developments in refugee status determination procedures in Europe, namely automated dialect recognition (Van Der Kist, 2025). In order to illustrate two points, I will present an example taken from a pilot project in the German context. First, how the design of this auditory technology orders the perceptions and actions of caseworkers in unequal relation with the asylum claimant. Secondly, how it redistributes who or what can (or cannot) be said or perceived in these administrative-legal procedures.

Making Linguistic Identities Perceptible and Actionable

In the aftermath of the so-called “migration crisis” in 2015, the German Federal Office for Migration and Refugees (BAMF) developed the Dialect Identification Assistant System (DIAS). This is a data-driven technology designed to assist authorities in processing asylum applications in a more efficient, secure and accurate manner (BAMF, 2018, 2022). At its core, DIAS employs speech analysis technology to assess whether the accents and dialects of asylum seekers align with their stated countries of origin. Similar to Van de Ven, I want to start by “unpacking” the automated dialect recognition device itself, unveiling how the identity of a migrant’s voice is rendered perceptible and actionable.

There are several intermediate steps between the registration and identification of the asylum seeker. The first step is to make the voice “machine readable” (van der Ploeg, 2005). By asking the asylum seeker to speak through a phone, a sensor scans their voice characteristics and generates a datafied version of the speech sample of it that is then stored in a local database from the asylum administration. The second step consists of the processing of this speech sample, where its features will be compared against templates stored in the database. The latter database consists of language samples of native speakers that had already been collected beforehand. Moreover, a model has been created based on machine learning, which is able to predict relations between them. The third and final step is the generation of an output, an electronic document with a comparison score measuring the degree of similarity between the speech sample and the template. For example, it may indicate that, according to the algorithmic assessment, there is a 54 percent confidence that the asylum seeker speaks Levantine Arabic.

It is through these many intermediate steps that the migrant’s voice—which is assumed to have unique biometric features—is being “made calculable” by caseworkers in the asylum administration. This enables a highly specific way of perceiving the asylum seeker’s identity that may have a significant impact on the outcome of their application. It is therefore crucial to acknowledge, as demonstrated by Van de Ven and Klonowska, that the process of identification entails a transformation—or “translation” (Callon et al., 1981)—whereby certain information is filtered and lost in order to gain these new insights (Bellanova and Fuster, 2019: 360). The final outcome of this process is contingent upon a multitude of factors that can influence the outcome and alter the final result (Callon and Law, 2005). For instance, asylum procedures are predicated on the highly contentious yet bureaucratically entrenched categorizations of identity, which in the case of automated dialect recognition are also those founded upon the assumptions of “dialect geography.” Embedded in the device is, therefore, the (methodological nationalist) presupposition that the language one speaks is tied to one’s place of origin. They are not sensitive to the wide variety and mutability of dialects. This assumption can become problematic in real asylum cases because, as Pfeifer (2023: 196) notes, “migrant biographies often involve spending months or years in transit, which means that people become immersed in different linguistic communities and contexts”.

Redistribution of the Sensible

The point here is not that automated dialect recognition system fails to represent asylum seeker identities. Rather, what interests me is that the sensing technology requires its own particular translations that pattern the perceptions and actions of decision-makers in ways that are somehow different from what came before. For instance, the perception of asylum seekers in conventional administrative-legal procedures is in large part determined by listening to the asylum story. Whilst it also depends on auditory faculties of the caseworker, this spoken, dialogic form of mediation diverges from those of automated dialect recognition (Van Der Kist, 2025). The automated dialect recognition systems are not interested in “what asylum speakers actually say” but are rather “grounded on the assumption that whatever they might say is not worth being taken into account before the trustworthiness of their belonging is ascertained” (Bellanova and Fuster, 2019: 360). Moreover, its calculation does not offer any understanding or explanation for the output. Rather, the algorithmic approach to speech simply suggests a correlation between the language model trained on an existing database and the speech data extracted from the asylum seeker. Other more complex and nuanced aspects of identity, including biographical markers, and how asylum seekers narrate them, are rendered irrelevant to the algorithm.

For the moment, automatic dialect recognition technologies are still in its pilot phase. Elaborating on Welfens’ contribution, in arguing that caseworkers’ security decisions are shaped by these sensing technologies is not to mean that they are going to be determined by it.⁷ As Scheel (2024) shows in his article, asylum administration in Germany is characterized by a reliance on a diverse array of techniques and technologies that shape the exercise of discretion. In this context, the implementation of the automated dialect recognition has, in effect, failed to capture the interest of caseworkers (Scheel, 2024: 8). The use of biometric sensing has been shown to frequently exceed the intentions of its designers, requiring constant “tinkering” to make it operational (Kloppenburg and Van Der Ploeg, 2020; Van Der Kist, 2024 see also Klonowska’s contribution). More importantly, migrants also have agency in resisting or circumventing these biometric control technologies (Scheel 2019). In short, not only do these examples illustrate how perceptions and actions are enabled and constrained by technologies (Gibson, 1979; Verbeek, 2005), they underline the need for situated analysis of whether and how these technologies are further taken up by their users or resisted by their subjects.

In extension, reflecting on the potential impacts of “sensing” in security studies should not just be concerned with the mediation of human perception by technological artifacts. I argue that it is equally, if not more important, to engage with what is rendered imperceptible by these machines (see also Brighenti, 2007). For instance, by carefully analyzing the affordances of sensory technologies like the automated dialect recognition system (amongst the many other technological devices that are currently being piloted and implemented) may bring into view the general shift in what can be said and heard in refugee status determination. This implicates the asylum administrator, whose perceptions are ordered by the mediation of the sensing technology, but even more so the asylum seeker, whose capacity to speak is reduced to a (meaningless) calculation (Lorenzini and Tazzioli, 2018). In short, by working in these different experiential registers, it will inevitably condition what might come to make sense in these administrative-legal domains: what bodies or voices can be rendered (in)visible, (un)sayable, or (in)audible (Rancière et al., 1999).

To this end, the conceptual and analytical potential of “sensing security” needs to be qualified further in terms of its political implications. For one thing, as the example of automated dialect recognition makes clear, its affordances echo a politics of control in which asylum seekers have been spoken for and silenced. In other words, the space for migrants to make their voices heard is, in part, limited by the technological constraints.⁸ Famously, Rancière et al. (1999) pointed to important moments in politics when those who “have no part” assert themselves as equal speaking beings—notably by interrupting the dominant perceptual regime. We may struggle to find these political moments in the examples of sensing security provided in this collective piece. But as I argue elsewhere (Van Der Kist, 2025), the problem with speech recognition technology is not only political but also legal: it revolves around the fact that claims to protection have traditionally been articulated through speech acts—mediated by the spoken or written word (Hildebrandt and Rouvroy, 2011). And it is precisely this ability to become audible, (visible and legible) as a political-legal subject, in close proximity to the sovereign power of the state, that is at stake in technologically-mediated practices of sensing security.

Methodology

For this article, we conducted eleven unstructured interviews with twelve professionals in computer vision in the field of security. The participants were purposively recruited based on their direct experience of and involvement in at least one of the following activities: (1) the design and development of computer vision algorithms and models; (2) the practical deployment, operational integration, and routine management of these systems in real-world security contexts; or (3) active critique, resistance, or contestation at a policy level of the implementations of computer vision technologies in contexts of (in)security.

Given practical constraints such as the specialised nature of this field and the limited pool of experts, we utilised convenience and snowball sampling techniques to recruit interviewees, primarily guided by participant availability, ease of access, and strategic relevance to the research question. Our selected interviewees should not be understood as statistically representative of a broader population or as ‘illustrative representatives’ of the field (Mol and Law, 2002: 16–17). Rather, through purposive sampling, interviewees were deliberately chosen to ensure variation across cultural and institutional affiliations, professional roles, and types of engagement with algorithmic practices.

The interviews were conducted in English in the Netherlands (6), Hungary (3), Germany (1) and Poland (1) by Ruben van de Ven and Ildikó Plájás, and subsequently by Clemens Baier. We employed a predominantly unstructured interviewing approach, initiating each conversation with two guiding prompts aimed at eliciting initial reflections. We began by asking the interviewees if they use diagrams in their daily practice. We then posed the prompt: ‘When we speak of ’security vision’, we speak of the use of computer vision in a security context. Can you explain, from your perspective, what these concepts mean and how they come together?’ Initial interviews revealed emergent thematic patterns, prompting the adaptation and refinement of subsequent interview questions to explicitly address these recurring themes.

At the practical level, we presented our interviewees with a large, A3-sized, digital tablet, and asked them to draw a diagram while answering our questions. Ruben van de Ven programmed an interface that could record conversation in drawing and audio. The participants could not delete or change their drawings, so their hesitations and corrections were preserved.¹⁰ Breaks appear in the conversation as the interviewees need to think about how to draw (Bravington and King, 2019: 509). Sometimes, the participants seemed uncomfortable about their ability to draw figuratively (see also Copeland and Agosto, 2012). These moments allowed us to clarify that our interest is not so much in the figurative as much as in the relations they drew. Some of the participants therefore opted to diagram with words, rather than figures.

Some of the diagrams that were drawn during the conversation show similarity with the kind of figures the practitioners work with in their own practices. Diagramming is indeed a key method in the field of technology, most notably in the conceptualisation and design of computational practices (Mackenzie, 2017). This can primarily be seen in the use of flow diagrams (Soon and Cox, 2020: 221), that represent different states or stages of computational processing as data ‘travels’ through a system. In our conversations, this style of drawing is particularly visible with those who develop or manage systems for algorithmic security vision. One interviewee explicitly mentioned he was reproducing his slideshow presentation in drawing as he went to explain some of the key concerns of his project. Many of the figures that appear in drawing thus do not exist purely for the conversation, but resemble or reproduce, both in form and content, the figures used by the practitioners in their daily practices. However, as the interviewees draw them on the tablet, they invite reflections that provided crucial insights. For example, one interviewee, a legal scholar and activist, drew a face-database, after which he reconsidered the visual representation: face-databases do not necessarily contain images of faces, but facial features in numeric form. While he initially drew on a common cultural representation of face recognition and surveillance, his technical expertise made him go against such a way of visualising these technologies (van de Ven and Plájás, 2022 in this bundle as Annex C). Thus, the interviewees tweak the images by drawing new relations, adding layers of explanation, and, as the conversation progresses they refer back to what they drew before, ‘hyperlinking’ the conversation back to earlier concerns.

In the phase of data analysis, the software we developed allows the diagrams to be annotated, creating short video clips, facilitating comparisons among the conversations. All participants except two provided explicit consent for identification by name¹¹. The multimodal tool was created and used to explore the possibilities offered by diagrams to elicit the main three affordances described earlier : (1) relational thinking; (2) graphical manipulation of abstraction; and (3) generative open-endedness in discussing the heterogeneous relations that constitute algorithmic security vision (see van de Ven and Plájás, 2022 for a more detailed account of the methodology and its affordances). Integrating temporal and dynamic aspects of diagrammatic practices was particularly significant for capturing not merely static visual representations, but also the performative processes through which these relational features emerge and are enacted. It is thus important to note that while the PDF version of this article contains only static image representations of the diagrams, the multimodal version allows accessing the full audio-visual excerpts of the diagrams¹². Our analysis refers to the audio-visual material.

1. From photographic to cinematic vision

The first finding of our diagrammatic interviews is a noticeable shift in the temporal dimension of the uses of computer vision in security technologies. With the advancements in algorithmic models and computational power, more systems than ever are able to assist security decisions based on video data. This shift from still images to video has implied an exponential growth in data which necessitates advanced infrastructure and analytical techniques capable of handling and interpreting continuous streams of visual information. As Gerwin van der Lugt notes, ‘Most cameras do about 30 frames a second. So if you have for example [in] Amsterdam 300 cameras [with] 30 frames per second for what is it? This is basically the number of frames generated each week. Yes, about 100 million’.

This, in our view, marks a shift from what we could call a ‘photographic’ mode of algorithmic analysis, such as biometric imaging (Pugliese, 2010), facial, iris and fingerprint recognition (Møhl, 2021), and body scanners (Leese, 2015). While the former type of systems works on the premise that it can identify individuals on the basis of unique and immutable features of their body (a fingerprint, an eye iris), the new, ‘cinematic’ systems work instead from the assumption that temporary anomalous states, grounded in suspicious movements or emotions, for instance, are what allows the identification of suspects, regardless of their intrinsic features. In the ‘cinematic’ mode of analysis, suspicion is thus not established only inside a single frame, but across a succession of them. This does not mean that both logics cannot be used simultaneously within an integrated system (for example looking in a crowd at movements of people previously identified through facial recognition), but rather that the ‘cinematic’ mode is a relatively new feature that comes with its own affordances and politics¹³.

Two diagrams represent this shift particularly well. Figure 2, by Ádám Remport¹⁴, illustrates the photographic mode. It depicts how photo snapshots produced by several surveillance cameras equipped with face recognition technology analysed over a longer period of time, can identify the same person visiting a bar, a church, an NGO. Remport spatialises the functioning of a facial recognition system, which constructs movement through the placing of cameras along the route of an observed individual. Here, the diagram quite literally maps the ways in which a series of static data points (images) can be used to infer and analyse movements and life habits over time and space.

Figure 3, by Gerwin van der Lugt¹⁵, shows how the alternative is ontologically different, through a different graphical spatialisation of the relations between the same entities. The system van der Lugt develops is meant to analyse the behaviour of people by looking specifically at movements and distinguishing different behavioural patterns. This system, as he explains, ‘can just detect things like punching, hitting, kicking, dragging someone or pushing someone and then combine those detections and at some point conclude that there is an ongoing violent incident or aggressive behaviour’.

The drawing on the top of Figure 3 represents the photographic paradigm: when processing video data, the frames (f₁, f₂, f₃) are passed to an object detector (YOLO), which runs independently on each frame, one after the other. In the second diagram, the video frames are no longer processed as individual frames but coalesced into a single data unit of 10 frames (f₁–f₁₀) that is passed to the algorithmic model (M). Multiple 10-frame units are then analysed by an additional process that looks at how these groups of frames relate to each other. This allows models to identify and categorise patterns across frames that happen in time, for example a hand gesture. By using the diagrammatic schemata to highlight the distinction between the two systems, the diagrams reveal that it is not a variation of the previous, photographic logic, but a profoundly different one.

This shift has political consequences in how models perform suspicion. The cinematic logic no longer relies on the inscription of supposed ‘indelible’ truths in biological features of individuals (faces, fingerprints, retinas); instead it operates by categorising unfolding movement as suspicion (a suspicious gesture, gait, facial expression), encoded across the frames. The advent of the cinematic dimension thus constructs suspicion as time-based, ephemeral and changing. Suspects no longer need to be categorically present in or absent from a space—it does not matter if they enter the NGO office—but bodies are marked by their movements through space, they can be passing, traversing, entering, or escaping. Thus, while earlier studies on biometric technologies have located the operational logic on identification, verification, authentication, in other words, knowing the individual (Ajana, 2013; Muller, 2010), the cinematic vision finds its operational logic in the mobility of embodied life (see Huysmans, 2022), and its governance through the dynamism of ever-changing ‘real-time’ clusters (Isin and Ruppert, 2020).

2. Synthetic data

A second major theme that emerged from our diagram-interviews is the growing significance of synthetic data in reshaping AI training pipelines which raises new ethical concerns. Over the past two decades, much of the critique of algorithmic security systems has centred on the tension between the ‘reality’ they claim to capture and the processes through which those realities are rendered computable. Central to this critique is Haggerty and Ericson’s notion of the ‘data double’ (2000), which highlights the constructed, arbitrary and always imperfect nature of data representations. This process of translation is indeed never neutral; it involves choices about what to include, what to exclude, and how to categorise, fundamentally shaping the reality that the system perceives and acts upon. Once encoded, these data profiles become the basis for social sorting (Gandy, 2021; Lyon, 2003), where realities are increasingly shaped by data-driven inferences and interventions, potentially leading to new forms of normativity and control (Bigo et al., 2019). The data used to train self-learning systems—which functions as ‘ground-truth’—is thus never a neutral or ‘raw’ representation of the world (Gitelman, 2013), but rather a product of selective encoding that risks perpetuating societal biases and representational shortcuts (Malevé, 2020).

Our research shows that synthetic data complicates these debates even more, appearing to actors both as a possible remedy and a further destabilising element to the question of representation of reality. In Figure 4, Sergei Miliaev¹⁶ describes the process through which an AI model is trained. The cycle begins with data acquisition, signalled by the words ‘data’, followed by ‘annotation’, ‘prototyping’, ‘deployment’, and an arrow back to ‘prototype; to emphasise the feedback-based refinements. The AI models are typically using real-world datasets, explains Miliaev in the interview, to generate inferences which can be acquired through ’web scraping’, by relying on ‘partner data’, and open-source footage (see the upper left corner of Figure 4).

Yet sourcing such real-world datasets—as shown in the controversies around large language models (LLMs) and copyright issues (Cyphert, 2024)—is a key challenge for the AI industry. Miliaev’s diagram allows us to pinpoint that this issue emerges from the very beginning of the AI training workflow; while hidden behind a sequence of processing steps—indicated by arrows in the diagram—the deployment of algorithmic security vision relies on a range of data sources, all of which raise particular concerns. Acquiring ethically sourced, high-quality data—especially for scenarios that require very specific data, like video—is increasingly difficult due to privacy laws and annotation costs. Moreover, these datasets—when they exist—either contain very little data, are of bad quality, or do not adhere to the requirements of the model.

To circumvent the challenges posed by the availability of reliable, varied and open access data Miliaev adds, thus introduces, another data source in the lower left corner of his diagram: synthetic data. The diagrammatic format allows to easily lay out the structure of relations graphically, evaluate it, and insert a new node (synthetic data) within the existing configuration. Synthetic data, which consists of computer-generated images created via 3D modelling, or by means of generative AI (genAI), can be a useful ‘workaround’, so explains Miliaev. Synthetic data enables the creation of large, labelled datasets tailored to specific scenarios, from suspicious activity to pre-crime behaviours. Crucially, because it is generated rather than captured, synthetic data bypasses some of the most stringent privacy restrictions. Furthermore, manipulated data can, for example, simulate diverse conditions—lighting, actions, faces—that are otherwise hard to capture at scale. Miliaev explicitly cites Microsoft’s work on training facial recognition models entirely on synthetic faces (Bae et al., 2023) as proof of this potential.

The adoption of synthetic data thus forces us to problematise the politics of ‘ground truthing’, i.e. the practice of establishing what the models should consider as ‘true’. One is fidelity: the challenge of ensuring synthetic scenes accurately reflect the variability and messiness of real-world environments. Poorly generated data—lacking realism in lighting, weather, or human behaviour—can lead to models that fail when deployed. Another issue is identity representation. While synthetic datasets allow designers to manipulate ethnicity, gender, age, and other attributes in a hope to de-bias the collection, Miliaev notes the difficulty of modelling long-term identity changes like ageing or behavioural variability. If the parameters used to generate this data are biased, the resulting models will inherit and even amplify social biases, as synthetic data does not escape the representational politics that shape captured data.

The diagram thus reveals how synthetic data destabilises conventional epistemologies. While conventional data sources for machine learning, such as operational data or web-scaping, have long been considered to provide access to a ‘raw’ or unfiltered truth, years of critique have established that data are never raw; they are constructed, interpreted, and made intelligible through a relational network of institutional, technical, and epistemic filters (e.g. Gitelman, 2013). This critique has now been taken up by many who develop algorithmic technologies, presenting itself as a problem in need of a technical solution. To this end, Miliaev annotates synthetic data with a moral assessment—‘fairness’—which is to be obtained by balancing various ‘attributes’ of the faces in the dataset different from what would be found in previously considered raw data. The new flow diagram that emerges no longer consists of merely technical description, but becomes indicative of moral and ethical assessments that lie at the heart of data gathering and processing. In what way, however, is this fairness established? Does the introduction of these debates imbue the diagram with ethical considerations, or is the ethics rendered as a technical concern? The ‘mean images’ of synthetic data—visual renderings of a non-existent, probable median—reproduce normative conceptions of reality, effectively complexifying the claims to indexicality of lens-based images (Steyerl, 2023). Instead of referencing empirical events, synthetic datasets rely on speculative associations and probabilistic reasoning, reframing our understanding of evidence, ‘ground truth,’ and credibility in machine learning practices.

3. Managing error: from the sublime to the risky algorithm

The question of the ‘ground truth’ is indissociable from the third central theme emerging from the diagrammatic engagements: the constitutive role of error in algorithmic security systems. While much literature addresses algorithmic prediction through self-learning systems (Azar et al., 2021), most of it has focussed on how these systems produce risky subjects (Amicelle et al., 2015; Amoore and De Goede, 2005; Aradau et al., 2008; Aradau and Blanke, 2018), based on environmental and individual features (Calhoun, 2023). In this critical body of literature, practitioners working with algorithmic technologies are often critiqued for understanding software as ‘sublime’ (e.g. Wilcox, 2017: 3), meaning that they are thought to believe that their systems ‘work’ in what they claim to do.

The diagrams we collected force us to rethink this premise. Models cannot be assessed through a binary criterion—working/not working – they are instead assessed through a series of error metrics. This is not a flaw that actors aim to fix once and for all; it is the asymptotic quest towards a complex mix of lowest error rates that drives the work of algorithmic security vision. Our diagrammatic interviews show both the importance of the error rates, but also the complexity to account for them and explain precisely what they are.

Figure 5, by CTO Gerwin van der Lugt is particularly enlightening in this regard. As he explains, the most prominent way in which error figures in model development is in its quantified form of the true positive rate (TPR) and false positive rate (FPR). Van der Lugt stresses the significance and definition of these metrics by marking them as the key variables to optimise, on the top right corner of the page (Diagram 5). Van der Lugt initially describes the FPR is the number of false positive classifications (FP) relative to the ‘input’, the number of video frames being analysed. When he continues to explain the TPR, he similarly marks it on the tablet as being equal to the number of true positive classifications (TP) over all inputs. As he goes on however, he realises that this is slightly incorrect. In fact, it should be ‘true incidents’. Looking back at the tablet again, he adjusts the original FPR equation: it should be specified as FP over the number of ‘false inputs’ (and not just all inputs). ‘These are quite tough to define but they are quite important, as these need to be really low’. The diagrams work here as an important support for van der Lugt to manipulate the abstract notions of errors through graphical inscription: by first noting them, then correcting them, the diagramming helps refine the explanation of a key yet complex notion to explain, even for the developer of the system. Moreover, as he corrects his definition, van der Lugt comments not only on the difficulty of these definitions, but also on the importance of having them right, as the these definitions determine the work of his development team, the ways in which his security operators engage with the technology, and how the system is evaluated by his clients.

As he goes on, van der Lugt ponders that algorithmic security vision is inherently prone to error, which affects the implementation of security practices. He illustrates this with Oddity.ai’s violence detection system, questioning whether playful fighting (stoeien) should be classified as violence. He argues for distinguishing between false positives—errors in evaluation—and errors in how the security problem is operationalised. He offers two reasons. The first reason is that excluding stoeien could reduce the algorithm’s TPR, disrupting the balance between TPR, FPR, and other performance metrics. The developers aim for fewer than 100 false positives per 100 million frames weekly, since too many alerts can desensitise human operators. The second reason is that excluding stoeien might introduce subtle biases—for instance, the algorithm might infer violence based on age rather than behaviour. Van der Lugt warns that such discrimination is both undesirable and hard to detect. In this view, error is not just mathematical but something to pre-emptively manage, placing responsibility on developers to anticipate and mitigate failure.

Another aspect concerns the mutable nature of threats and susceptibility to adversarial attacks. András Lukács¹⁷ notes that detection models ‘can be cheated by quite simple tricks’. Even data-rich systems remain vulnerable to intentional evasion, highlighting the challenge of maintaining relevance in evolving threat landscapes. John Riemen¹⁸ adds that poor input quality undermines facial recognition—for instance, the wrong still from a video may fail to identify correctly.

Practitioners suggest several strategies to respond to these issues. One is narrowing the algorithm’s scope to high-impact, clearly defined crimes. Guido Delver¹⁹ recounts moving from detecting vague ‘suspicious behaviour’ to focusing on burglary. Van der Lugt’s firm similarly focuses on weapons, vandalism, and physical violence. This narrowing reduces ambiguous boundary cases—both an ontological and an operational shift. Another strategy is continuous retraining. Dirk Herzbach²⁰ explains how operators feed annotated alerts back into the system. Parameter tuning to balance false positives and negatives is also key. This balance is political: as Jeroen van Rest²¹ notes, communities vary in their tolerance for error—some may accept high false positives for safety, others may not. Thus, the acceptable error threshold becomes a negotiation site between developers, publics, and institutions.

In this light, error is not external to algorithmic systems—it is their condition of possibility. As Pasquinelli (2019) puts it, machine learning is based on ‘formulas for error correction’; error is not a failure to remove, but the mechanism through which learning occurs. Amoore (2019) similarly notes that ‘it is precisely through these variations that the algorithm learns what to do.’ Error, initially a mathematical concept, permeates every discussion of algorithmic security vision. What emerges is a double articulation of risk. While risk is usually treated as external to, and produced by, security technologies, technologies of algorithmic security vision themselves become objects of risk management. The inevitable errors of machine learning algorithms place risk not outside, but at the core of the security apparatus.

4. Reconfiguring the human-in-the-loop

The understanding of algorithmic security vision as a practice of managing error has significant consequences for the role of the human within these assemblages—our fourth main theme. The critique of algorithmic security has often construed the human-in-the-loop as one of the last lines of defence to the inevitable erroneous outcomes of automated systems (Markoff, 2020). However, critical security studies have questioned this representation, emphasising the visualisation of algorithmic predictions in graphical interfaces (Aradau and Blanke, 2018) and how the operator's embodied decision-making is intertwined with the algorithmic system of which they are part (Wilcox, 2017). Furthermore, operators may find themselves uncertain about the system's functioning when errors occur (Møhl, 2021). In other words, when considering questions of responsibility, a system's operator cannot be seen as distinct from the algorithmic assemblage (cf. Hoijtink and Leese, 2019a). The question in a system's design thus is not whether a human operator makes autonomous decisions but rather, how they are to interface algorithmic predictions, and how issues of agency and responsibility are negotiated. The diagrams illustrate how this leads to differentiated processes of (in)visibilisation.

Looking across the various diagrams reveals a spectrum of system designs concerning the autonomy of security operators and the technical expertise they are expected to bear. In a first kind of designs, the human operator is central to the decision-making processes, acting as the interface between, and external to, algorithmic systems and surveillance practices. For example, Dirk Herzbach explains that when the Mannheim police is alerted to an incident by the system, it is the operator who decides whether to deploy a patrol car. Here, the human-in-the-loop has full agency and responsibility for operating the (in)security assemblage, with the capacity to evaluate and utilise algorithmic systems selectively and with care. Herzbach therefore prefers an operator to have patrol experience, so they can best assess which situations require intervention. He is concerned that knowledge about algorithmic biases might interfere with such decisions. In other words, the security operator is considered the expert of the subject of security and is expected to make decisions independently from the information that the algorithmic system provides.

However, in a second kind of design, the human operator is considered an integral part of the algorithmic security vision system, influencing and influenced by its operation. Such designs acknowledge that it might be more complex for the human-in-the-loop to perform as the primary counter to algorithmic error. This is illustrated by Guido Delver, project manager for the Burglary-Free Neighbourhood project in Rotterdam, which incorporates autonomous systems into street lamps. In Figure 6, Delver maps out the different ‘stakeholders’ of the project on a flat canvas, using the diagram as a way to spatialise relations of a heterogeneous set of key entities. He envisions the neighbourhood as a site where the public and private space—the residents and government—meet, and which also involves suppliers of technology and research institutions. By rethinking the traditional top-down hierarchy of the surveillance apparatus, Delver aims to counter government hegemony. Delver uses this rendition of the various stakeholders to ask the question ‘who [of these parties] owns the data?’. However, upsetting the traditional hierarchies comes with its own risks. Delver illustrates his point with a scenario in which the algorithmic signalling of a potential burglar may have dangerous consequences: ‘Does it evoke the wrong behaviour from the citizen? [They could] go out with a bat and look for the guy who has done nothing [because] it was a false positive.’ In this case, the worry is that the erroneous predictions will not be questioned. Thus, while often deemed to be ultimately responsible for the judgement that follows from the algorithmic process, human participation or ‘interference’ in the operation can likewise be rendered as harmful.

As a means to mitigate these concerns, in Delver’s project, the goal was to actualise an automated decision system (ADS), ‘with as little interference as possible’. Placed in the centre of his drawing, different stakeholders interact with this system in different ways. Thus, figuring the human-in-the-loop—whether police officer or neighbourhood resident—as risky can lead to the relegation of direct human intervention. For Delver, access to the data—both the data that makes inferences possible, as well as the outcomes produced by the device—needs to be curated.

The curation of data flows—acknowledging the fallibility of human decision making—is even more prominent in another diagram. In Figure 7, John Riemen (dis)assembles the process of facial recognition as it is employed by the Dutch police. After drawing three human figures, he stresses the relations among these multiple humans-in-the-loop, and how some data should be allowed to pass (indicated by the arrows in the diagram; for example, from a database expert who performs the search to an investigator of the case), while other flows of information should be blocked. For facial detection to be accurate, the human interaction with the algorithm needs to be sanitised of personal information.

By drawing the process as series of actors, all of whom are biased, Riemen sketches out how they are embedded in, and thereby influenced by, a set of mutual relations that needs to be regulated, here the diagrams capture the relational and sequential nature of these links.

The diagrams thus challenge the simplistic view of the human-in-the-loop as a fail-safe against algorithmic errors by highlighting the complex network of sociotechnical relations they are enmeshed in. For both Delver and Riemen, the inherent error—whether performed by a person or a computer—has to be mitigated by curating the transfer of information from one node to the next. More than multiplying the number of humans-in-the-loop, these diagrams encourage us to think in terms of ‘curatorial pipelines’ (Malevé, 2023) that mediate the different human and algorithmic operations.

5. Delegating accountability to benchmarks

The final theme of our diagrams concerns the question of accountability. Literature on the ethical and political effects of algorithmic vision has notoriously focussed on the distribution of errors, raising questions of ethnic and racial bias (e.g. Buolamwini and Gebru, 2018). This critique has now been internalised in the field of AI development; virtually all of our correspondents acknowledge the concern with bias.

To mitigate it, many AI developers—including some of our respondents—have come to rely on an external reference against which the error is measured: benchmarks. John Riemen, for example, who is responsible for the use of forensic facial recognition technology at the Centre for Biometrics of the Dutch police, describes how their choice of software is driven by a public tender that demands a ‘top-10’ score on a benchmark for facial recognition vendors, maintained by the American National Institute of Standards and Technology. Often shortened to the acronym NIST (e.g., Figure 7), this benchmark ranks facial recognition technologies of different companies by their error metric across groups. The mitigation of bias is thus outsourced to an external, and in this case US-based, institution.

By examining where John Riemen’s diagram ends (see Figure 7)—that is, where entities are no longer broken up into its component parts, but rather bracketed and considered as a whole—we get a sense of how issues are ‘black boxed’. As Riemen describes the process of facial recognition, he is able to explain and draw it in much detail. Therefore, when asked about bias mitigations, he is able to precisely locate the various measures of filtering that take place, drawing the ‘walls’ in between the different people (see the vertical line in the diagram). On the lower part of the drawing, the NIST database does not have the same detail in process, nor scrutiny. In Riemen’s diagram it becomes a single entity, represented by an acronym. As a benchmark, the facial recognition vendor test is implicitly assumed to have already implemented the measures that are required for its use by the Dutch police. The diagrams here provide a graphical illustration of the abstraction and erasure of particular processes, showing only the inputs and outputs within a general network. Put in more stark terms, where the diagram stops, the respondent’s responsibility stops. With NIST, the prevention of algorithmic bias comes to rely on a single benchmark: a de-facto standard that is managed by an external institution.

The problem is, while a particular kind of algorithmic bias (i.e., ‘demographic differentials’) is rendered central to the NIST benchmark, the mobilisation of this reference obfuscates questions on how that metric was achieved. For example, the NIST benchmark datasets are known to include faces of wounded people (Keyes, 2019). Moreover, questions about training data are invisibilised, even though that data is a known site of contestation; the Clearview company is known to use images scraped illegally from social media, and IBM uses a dataset that is likely in violation of European GDPR legislation (Bommasani et al., 2022: 154). Pasquinelli (2019) argues that machine learning models ultimately act as data compressors: enfolding and operationalising imagery of which the terms of acquisition are invisibilised.

This attention to the invisibilisation of and in/between processes reveals a discrepancy between the developers and the implementers of these technologies. On the one hand, the developers we interviewed expressed concerns about how their training data is constituted to gain a maximum FPR/TPR ratio; while showing concern for the legality of the data they use to train their algorithms—propelling for example the use of generative AI (see section 2). On the other hand, questions about the constitution of the dataset have been largely absent from our conversations with those who implement software that relies on models trained with such data. Occasionally this knowledge was considered part of the developers' intellectual property that had to be kept a trade secret. In such cases, for the implementers, a high score on the benchmark is enough to pass questions of fairness and to forgo any further inquiry into the inequalities propagated by the algorithmic system and the data through which it learns, thus legitimising its use. While the deployment of algorithmic models indirectly relies on the training data, it is not deemed relevant in the consideration for one particular model over another.

The configuration of algorithmic vision’s bias across a complex network of fragmented locations and actors, from the dataset, to the algorithm, to the benchmark institution reveals the selective processes of (in)visibilisation. It illustrates well how accountability is bracketed through the invisibilisation of the dataset as it is ‘compressed’, in Pasquinelli’s terms, into an algorithmic detection model, with the formalisation of guiding metrics into a benchmark. One does not need to know how outcomes are produced, as long as the benchmarks are in order.

Line 3: predict() the now

The object tracker describes the trajectories of those who passed by the camera, yet paradoxically, even before the detections are presented to it, it invokes the forward-looking logic of prediction. Those familiar with coding might wonder why the tracker needs two function calls. Why isn’t a single tracker.update(detections) function sufficient? Examining the operations that come into play in calling the tracker’s predict() function can reveal the entangled temporalities in the monitoring of movement. This means we need to step away from self-written code, to the code as written in the DeepSORT implementation (lst. 1).

def predict(self):
    """Propagate track state distributions one time step forward.
    This function should be called once every time step, before `update`.
    """
    for track in self.tracks:
        track.predict(self.kf)

In this first snippet, the discontinuity of video is immediately apparent. The two lines of documentation that are part of lst. 1 (indicated by """), describe when the predict function should be used (‘before update’) and stress how time is punctured as it is made to pass in a sequence of steps. This resonates with what Deleuze (2024), following Bergson, discussed on cinema: the equidistant perforations on the film-strip produced a uniform time interval between frames. Each frame of the video is an ‘immobile [section] of time’ from which movement is to be reconstituted (Deleuze, 2024). The tracker invokes yet another, nested, predict() function for all the tracks that it has at that moment in memory (the self.tracks variable refers to data kept in memory by the tracker; its present state). This suggests some information is carried over from the previous time step to the current, for which the predict() call is executed. Thus, to overcome discontinuity and stillness, the tracker has to ‘propagate’ previous measurements into the now.

def predict(self, kf):
    """Propagate the state distribution to the current time step using a
        Kalman filter prediction step.
    […]
    """
    self.mean, self.covariance = kf.predict(self.mean, self.covariance)

The origin of the tracker’s predictive logic becomes apparent when tracing the code invoked by track.predict() (lst. 1 line 55). The function’s definition can be found in another source file (see 2). The comments in the code refer to the Kalman filter. This technique, devised for the Apollo mission (Kalman, 1960), is one of the most used signal processing techniques in control theory to process discrete time-series data (Musoff and Zarchan, 2009). It is used not only in camera security, but also ‘to control a vast array of consumer, health, commercial, and defense products.’ (Grewal and Andrews, 2010). It integrates multiple signals from GPS satellites into a single location and velocity. It can estimate the movement of objects in the interval between individual radar detections. Missile defence systems use it to anticipate a projectile’s trajectory. In the case of the movement tracker, the distinct measurements that come from the object detector represent the position of a body. They do not contain the essential data on its velocity or acceleration that describe a body’s movement. The Kalman filter can mathematically infer these properties by subsuming the measurements of position into a predictive model of reality. The multi-object tracker overcomes discontinuous time by constructing the present via a simulation that ‘stretches forward’ observations across the temporal gap between frames. In other words, past tracker data is used to predict the now.

The Kalman filter’s predictive process is developed to operate in an inherently noisy system. The filter thereby operationalises Shannon’s mathematical theory of information, which states that any input value is distorted by aspects of reality one does not want to measure (Shannon, 1948). In case of the object tracker, the readings of an image sensor contain electrical noise, the compression of the image pixels by means of a video codec alters their values, and many more such distortions are bound to happen for each processing step. Likewise, the detection algorithms that locate the body in the still frame are assumed to be imprecise as they are, for example, impacted by occlusions. In a noisy system, whether data comes from an object detector, or any other sensor, no measurement can be considered absolutely certain. Therefore, while reality can be measured, due to the noise in the process, it is presumably ‘hidden’ from observation.³⁰ To come to terms with these uncertainties, the logic of measurement is shifted. In this context, although reality is hidden or unobservable, it is ‘latent’. In medicine, the latent period of an infection is the time exposure and symptoms, or between drug intake and effect (Veel, 2021). Similarly, a latent reality and its movements are dormant: they are still undeveloped, but bound to develop. The assumption then is that these innate developments of the latent subject can be modelled.

If we thus understand the predictive process in light of these core tenets of signal processing, the Kalman filter filters the measurements for ‘relevant’ data – the signal – by quantifying the noise in the system. Therefore, when the Kalman filter predicts the body’s position, and its first derivative over time – velocity – it does so as a multivariate probability distribution. The prediction covers a range of likely next positions centred on the mean, the spread of which depends on the modelled certainty, the covariance. By simulating the body’s most likely movements, the temporal logic of the forecast is invoked to anticipate the most likely next step, simultaneously quantifying the confidence of that prediction.

What is the role of this prediction in the broader functioning of the tracker? As the earlier snippet of code (lst. 1) shows, the predict() function does not return its predicted values for use by the end-user of the tracker (i.e. the programmer). Rather, the predicted present stays within the logical confines of the multi-object tracker’s tracks. This implies the forecast – the anticipated present – needs to be understood in the recurrent operation of the tracker, in which the same steps repeat for one frame after the other. In this operation, which runs for sheer infinite times when seen from the micro-temporal scale at which it takes place, the predicted present serves as a predisposition that shapes how new measurements are incorporated in an additional computational step.

Line 4: update() and the recurrent integration of state

The function of the predicted present thus becomes apparent in the subsequent step of the tracker, as it is fed the detection data of the latest frame. Here, the predicted present provides the ground upon which this incoming data is assessed; the result of this assessment becomes the input for the next prediction. To that end, the update(detections) of the multi-object tracker performs two main tasks. A call to match() associates the incoming data with existing tracks. After associating, a call is made to the second part of that track’s Kalman filter in another, nested, update() function. The Kalman filter’s update integrates the associated data into the track state (see lst. 3), by which the next timestep’s prediction will be made. It is in the recurrent integration of state that a different formulation of movement becomes apparent; one that is anchored in the subject’s temporal variability.

def update(self, detections):
  …

  # Run matching cascade.
  matches, unmatched_tracks, unmatched_detections = \
      self._match(detections)

  # Update track set.
  for track_idx, detection_idx in matches:
      self.tracks[track_idx].update(
          self.kf, detections[detection_idx])

  …

In match() (see lst. 3 line 98), the tracker associates new measurement data (detections) with previously processed tracks based on their movements. With the original SORT technique, the predicted present positions are compared with the newly detected positions based on a distance metric. The detections are associated depending on which track is nearest in the two-dimensional image space (the Hungarian algorithm). If a position cannot be associated, a new track instance is created; if a track has no matching detection, it is marked ‘lost’. With DeepSORT, a feature vector augments the two-dimensional distance metric. This vector is a numeric description of a small portion of the video frame, obtained by cropping it to the detected bounding box and feeding that to a (deep) artificial neural network³¹. The similarity of feature vectors is again determined using a distance metric. However, the vectors involved in the comparison arise from distinct temporal logics: a measured now, and a predicted now, which was modelled after past measurements. As the predicted now is not merely a point, but a distribution, the distance metric is not purely geometric but reconciles the statistical likelihood of the measured position. Moreover, to associate data with subjects the tracker thus does not rely on a body’s static identifiers, such as fingerprints, or faces. Rather, the value of this feature is updated on each iteration of the tracker. In the recurrent matching, the identifying features of the moving body change as the movement unfolds.

The subsequent Kalman update() function recurrently integrates measurement data into the tracker’s state (see lst. 4). The quantisation of movement into distinct frames, and the noise inherent in these observations, poses a computational challenge: any prediction of a body’s position based solely on the last measurement is considered highly unreliable. However, the Kalman filter cannot simply accumulate all previous data points to run its calculations. Multi-object trackers are made to run on ‘edge’ devices (rather than central cloud servers), such as cameras or embedded computers, thus both memory and computing power are limited. Retaining the full history of an object for the prediction of their next position would both require an increasing amount of memory and exponentially lengthen the amount of time needed for the prediction. This increase in computational complexity impedes the low delays upon which the tracker’s ‘nowness’ relies. To overcome these limits, the process of integration compresses the sequence of observations, each of which merely captures a body’s still position, into a behavioural model of the moving subject (cf. Pasquinelli and Joler, 2020a). As this state should not represent how things are, but what they most likely will become in future timesteps, the Kalman filter augments the measurements with the derived velocity. Like the latent spaces in more complex generative neural networks, the filter’s integrated state fulfils ‘a seductive political promise of locating the hidden tendencies in populations, places or scenes’ (Amoore et al., 2024: 8). Thereby, as data comes in, the integrated state shapes incoming data; and, as the state is used to predict the subsequent timestep, it is itself shaped by that data.

def update(self, mean, covariance, measurement):
    […]

    projected_mean, projected_cov = self.project(mean, covariance)

    chol_factor, lower = scipy.linalg.cho_factor(
        projected_cov, lower=True, check_finite=False
    )
    kalman_gain = scipy.linalg.cho_solve(
        (chol_factor, lower),
        np.dot(covariance, self._update_mat.T).T,
        check_finite=False,
    ).T
    innovation = measurement - projected_mean

    new_mean = mean + np.dot(innovation, kalman_gain.T)
    new_covariance = covariance - np.linalg.multi_dot(
        (kalman_gain, projected_cov, kalman_gain.T)
    )
    return new_mean, new_covariance

Where in the predict() step the confidence of the prediction is assessed, the update() function – which in the recurrent operation both follows and precedes predict() – is to quantify the degree to which the body’s movements adhere to its derived model. As outlined in the previous section, the information theories that underlie the Kalman filter assume noise to be present in the measurement. Moreover, the Kalman filter’s model, compressing all observations into a single representation, is assumed to be lacking. Consequently, the approximated movements of the body are never exact. A metric central to information theory, surprise (or, ‘information content’, denoted as innovation in the code, lst. 4 line 192) describes this difference between the predicted present and the observation data. The Kalman filter’s update() step uses this difference, the surprise yielded by the coming together of measurement and prediction, to adapt the weighing (the kalman_gain) of the simulated positions versus the latest measurement data (Musoff and Zarchan, 2009). By recurrently tweaking the mean and covariance, the filter minimises the expected surprise of future measurements by minimising the statistical difference between what was expected and what was observed. Measuring conformity of the movements to its model, surprise operationalises the gap between the heterogeneous temporal logics of the predicted present and the immediate observation.

After the integration of state, the cycle resumes. All that is carried over to the next iteration of predict() and update() is the inferred present state and its uncertainty. In this recurrent process, gathered knowledge is assessed by its virtue to predict subsequent positions, while the subject is assessed by its adherence to the model. In the prediction of the now, the moving subject, temporal variable and fundamentally unknowable, is governed as a signal.

Spaces that order relations

First, the diagrams use space to order concepts and relations. For example, in one of the interviews, when Guido Delver discussed the “stakeholders” of the project he managed, he did not list them, but placed them instead on two axes: municipality/police ↔︎ residents and industry/suppliers ↔︎ research/universities. All these parties “gathered around” the public space in which computer vision was deployed. During another interview, András Lukács used bullet points with written concepts, but instead of placing these vertically, he placed these elements between two extremes: “security” and “computer vision”. In this drawing, security vision emerges in the centre of the image, where the two extremes overlap (figure 16, Top). On another occasion, Gerwin van der Lugt more explicitly drew a Venn diagram to locate his expertise on computer vision in a particular subset of the field (figure 17).

In the space that emerges, the placement of various concepts helps to indicate what differentiates and what unites the object of research. Through these spatialisations, we learn that the relations between the entities mentioned in the interviews cannot be reduced to either connection (as would be signalled by a line in a network visualisation) or containment (as in a Venn diagram). They are much more complex. Sometimes connections are assumed but left implicit, while at other times they are signalled only by bringing two entities into physical proximity but never spelling the connections out. Connections are made explicit only when they figure in a specific story line.

Multiple configurations

The second way in which space matters in the diagrams is to allow for multiplicity within the drawings. Most drawings, while forming a whole within the context of the conversation, can also be seen as being composed of many distinct drawings that are the results of loosely connected topics discussed by the interviewees. These distinct drawings appear side by side, sometimes even curving around one another, ever shifting in scale. Scribbling asides in the corner, the interviewees often tried to squeeze as much as possible within the boundaries of the 24”-inch canvas. The equations in figure 17, for example, were so squeezed in that they had to be explicitly demarcated from the rest of the drawing by a line. Even though the interface allows for infinite dragging and is theoretically unbound, the thick black borders of the pen display did in fact matter in shaping the drawing, as the drawings try to take up the space that is left available to them. The absence of a uniform projection liberates these multiple drawings-within-a-drawing from a mutual visual hierarchy. While this might make the reading of the diagrams difficult, it allows the diagrams to bring together concepts and visual language from across various incompatible registers. They appear not as coherent narratives, but as collections of figures, thoughts and associations, summarising and synthesising larger ideas that hang together by virtue of their mutual appearance in the diagram.

By allowing for both order and multiplicity, the diagrams con-figure incompatible concepts and narratives. Moreover, the spatialisation of the conversation cannot be seen as distinct from the diagram’s temporal dimension. During the interviews, the drawings often became a visual referent that facilitated further elaboration and explanation. In these moments, the strokes on the canvas provided landmarks for the conversation. This becomes apparent within the conversation as the interviewees turn to the drawing to point out what they are speaking about or to pick up the conversation from a particular point. The drawings also served as visual references in the phase of their analysis. After we had conducted the interviews, we printed out the drawings on A3 paper and hung them on our office walls. While we thus temporarily flattened the diagrams, removing their temporal dimension, it was by looking at the printouts that we could retrace the conversation and recall the topic being discussed with each specific shape. The diagrams again spatialised the conversations, this time those we were having among ourselves in the office.

In the next section, we will elaborate how the diagrams, through both their temporal unfolding as well as their mutually coming together, foreground the ways in which security vision con-figures uncertainties and hesitation.