‘Have the kids come home?’: How open-source technologies are reshaping conflict and humanitarian operations

Researchers affiliated with the nascent, U.S. government-backed Conflict Observatory have been capturing and analyzing publicly and commercially available data and digital imagery since the weeks leading up to Russia’s full-scale invasion of Ukraine in 2022, to help spotlight possible war crimes and mass atrocities in that part of the world. 

Over the course of the war so far, the group has used open-source technologies to supply dozens of intelligence reports with evidence of Russia-led efforts to force passportization on Ukrainians in occupied areas, destroy the nation’s cultural heritage sites and major medical facilities, and — among other international violations — force the relocation of thousands of Ukrainian children without consent from their parents. 

Historically, “in most cases, the victims of the things we document are dead. Here, in the children’s case — it’s the rarest thing I’ve seen in my career — the victims are alive. They’re kids and they’re in a hostage situation,” human rights investigator Nathaniel “Natty” Raymond told DefenseScoop in a recent interview.

Known for his demonstrated expertise tapping open-source capabilities to uncover war crimes and devastation around the globe over the last nearly two decades, Raymond now serves as executive director of Yale University’s Humanitarian Research Lab and a key Conflict Observatory researcher. The State Department’s Bureau of Conflict and Stabilization Operations officially unveiled the hub in May 2022, as an independent project it was supporting in collaboration with Esri, Yale’s Humanitarian Research Lab, the Smithsonian Cultural Rescue Initiative, PlanetScape Ai, and others.

Since then, Raymond and his colleagues have generated a platform to steadily and widely disseminate records and assessments regarding Russia-perpetrated war crimes, and — separately, but using similar methods and tools — an asset to remotely monitor intensifying conflict activities in Sudan.

In a recent briefing with DefenseScoop, he discussed how the U.S. government is shifting into a future where decision-support information and intelligence operations are largely “about leveraging open source to develop facts that can affect actions and outcomes.”

In the earliest days of government satellite imagery, data was primarily viewed “by the president and like three to four other people,” Raymond noted — whereas now, people have an expectation to see things that were once limited to the nation’s most secretive observation capabilities.

He shed light on how the Conflict Observatory’s unfolding research and measurable impacts demonstrate ways open-source technology is beginning to reshape military and humanitarian operations in real time. 

“This doesn’t change everything, but it changes a lot. And it’s not just about human rights and human security, it’s about the increasingly forward-facing, public-facing roles of technologies and tradecraft that were once only for classified spaces,” Raymond said.

‘They’re taking selfies’

“The Conflict Observatory provides a platform to independently document, verify, and disseminate open-source information on likely human rights abuses, war crimes, and other atrocities in countries of interest. This information is collected and preserved consistent with international standards for use in ongoing and future accountability efforts, including potential civil and criminal legal processes,” a State Department spokesperson told DefenseScoop on Thursday.

The nascent organization analyzes what the official noted State views as “reliable data,” to verify reports from other sources, like the media, first-hand accounts, and field reports.

“We are engaging with a broad range of efforts to preserve and document evidence of potential war crimes and other atrocities. In the case of Ukraine, this includes inquiries carried out by the United Nations, the Government of Ukraine, the expert missions established under the Moscow Mechanism of the Organization for Security and Cooperation in Europe, accountability-focused civil society organizations, and other international partners,” the spokesperson said in response to DefenseScoop’s questions regarding the agency’s application of such technologies.

Open-source intelligence, or OSINT, is generated via the capture, preservation, evaluation and analysis of publicly available raw data and information to answer specific questions. 

Nowadays, the term “open source is thrown around all the time — to the point where it’s like artificial intelligence — you could be talking about one thing, you could be talking about 30 things, you could be talking about nothing,” Raymond said.

“The way I define it is that there’s sort of four baskets of things that end up being relevant to open source,” he explained.

The first basket encompasses anything that any person can find on the internet. The second, in his view, is commercial data that not everyone can find on the internet — but that can be purchased or acquired often under licenses.

The third basket is “heuristic information, which is information about network functions,” Raymond said. Essentially, this is informatics or information about how records and data are processed, transmitted and stored.

The fourth basket — algorithms — builds on the other three. As researchers can track how all the information moves, from there they can use algorithms “to get from noise to signal quicker,” Raymond noted, and also comb through it all and provide needed insights at a quicker pace. 

“Often, having those algorithmic capacities itself allows insights from basket one that you wouldn’t have if you just had basket one. And so, really, it’s a workflow — everything on the internet to everything you can buy and license, to then everything you can begin to build sets around to understand when, where, how and why it moves. And then moving to [basket] four, which is then how can you build information filters and fusers to then accelerate your collection-to-analysis-to-action cycle — that’s the trick,” Raymond told DefenseScoop.

In addition to leading the Yale School of Public Health’s Humanitarian Research Lab, Raymond is also a lecturer in the Department of the Epidemiology of Microbial Diseases. His research interests concentrate on health implications of the forced displacement of humans; methodologies for assessing large-scale disasters; and the human rights and security implications of information communication technologies for vulnerable populations, and specifically in armed conflict.

He’s known in the field for helping pioneer the application of satellite imaging and remote sensing to spotlight deteriorating humanitarian conditions and record evidence associated with alleged mass atrocities and crimes against humanity. 

Building on that career, Raymond was teaching at Yale for about four years when he said a good friend and former colleague from State called him up in early 2022 to talk about the need for a means to preserve and use publicly and commercially accessible information and records as evidence of possible future war crimes in Ukraine.

“I felt like Paul Revere level — hopping on the horse and going to Lexington and Concord — because, even before the attack started, we were moving into position to be ready,” Raymond said. 

The team of experts he was working with shared a “critical two weeks at the start of February” right before Russia’s invasion, he said, which involved identifying proper personnel systems and data sources and developing workflows. From there, they started making “assumptions, which turned out to be right,” he noted, about what the nature of the violence was going to look like, against what the confounding variable factors were going to be in the imagery and data availability. 

“We made a decision that our first focus was going to be on attacks on hospitals. And we thought that was going to be one of the critical battle fronts — which it turned out to be, with Mariupol — so we developed the base layer, and we started trying to get as much imagery to do change detection as possible,” Raymond explained.

That initial phase was particularly difficult because it played out during a harsh winter, with significant cloud cover and snow — causing a lack of baseline imagery in certain critical places they needed to see how things were going to change over time. 

“The way I describe it, from February and April, is it felt like the geospatial equivalent that we were Spitfire pilots in the Battle of Britain, where they slept with their boots on in the cockpit. There was a point where within I think it was four to six weeks … we did over 250 [battle damage assessments] in 20 operational days. That meant we were at a point where, across the team, we were doing artillery damage assessments, fire damage assessments on an average of almost like 30 to 45 minutes to target. So it was Battle of Britain-like, I think. Obviously I don’t know within [the National Geospatial-Intelligence Agency and National Reconnaissance Office] how fast they’ve gone — but it is manually the fastest BDA tempo I’ve ever seen in my life,” Raymond told DefenseScoop.

The Conflict Observatory uses commercial imagery data from Maxar, BlackSky, and PlanetScape Ai and others, as well as commercial synthetic aperture radar data, produced from sensors that give off energy and then document the amount of that energy reflected back after engaging with Earth. This essentially provides the capability for humans to “see through” dense clouds. 

Early on, the observatory was leaning heavily on Maxar’s capabilities, but since then — across Sudan and Ukraine efforts — the team is “now ‘playing the orchestra more’ and being able to do tip and cue,” Raymond said.

In its most recent publication on Ukraine, the hub presents “evidence of systematic and widespread attacks on medical facilities in Mariupol during Russia’s invasion of Ukraine” from Feb. 24 to May 20, 2022. 

Researchers geolocated all medical facilities in Mariupol and cross-corroborated them in a minimum of two publicly available databases. Two or more analysts then conducted pre- and post–satellite imagery damage assessment — ultimately finding that 77% of buildings providing medical care sustained damage in that siege. Building sizes were not correlated with damage, the analysts noted, which they said suggest “the targeting of medical facilities or indiscriminate, ubiquitous destruction.”

“This is the first evidence of the scale of alleged war crimes in an ongoing conflict,” the researchers wrote.

In prior reports on devastation in Ukraine, the Conflict Observatory has also uncovered evidence of damage and looting to more than 1,680 Ukrainian cultural heritage sites; discoveries of mass, unmarked graves at Pishchanske Cemetery in Izyum after the city’s liberation from Russian control; a decrease of Kyiv’s nighttime light production following a wave of missile and drone strikes; and a Russian filtration system to register and detain Ukrainian citizens in the eastern area of Donetsk Oblast — in that case violating international humanitarian law.

“Our most well-known report since we started on both Sudan and Ukraine is the children’s report from February of this year, which helped catalyze the indictment of [Russian President Vladimir] Putin and [Russia’s Commissioner for Children’s Rights] Maria Lvova-Belova on two war crimes — charges of illegal transfer and deportation of Ukrainian children,” Raymond said.

His team asserted in their report that they confirmed at least 6,000 Ukrainian children detained by the Russian government and identified a network of more than 40 stations where toddlers, kids and teenagers were being held in custody.

“Imagery was involved in that, but it was a significantly junior player, compared to OSINT,” Raymond explained.

The work was essentially what he called “a synthesis of social media data” from various apps, combined with photogrammetry analysis of selfies of Russian officials. Satellite imagery was helpful in geolocation, but it was a supporting character to OSINT in ways it could not have been in certain places perhaps in Africa, where the digital infrastructure is much different.

Raymond said his team knew they wanted to analyze evidence that could inform allegations for a report on what it deemed Russia’s systematic program for the re-education and adoption of Ukrainian children. 

“We were banging our head against the wall until our head went through the wall,” he said.

Experts did not feel that the data they had access to was ready for application in the summer of 2022, but by the fall, they realized there were Russian officials on the local level, Raymond noted, who “appeared to be the heart of the chain of command implementing this program — and they’re taking selfies.”

His team used those selfies to geo-locate an associated logistics network and network of camps holding young Ukrainians. 

“Then we realized, wait a second, each of those transfer acts is a unique offload of children that actually helps us control for redundancy. Then we started to count the numbers, and that got us to the 6,000 number — which we know is low — but we got that by beginning to aggregate the open source where it was the perpetrator themselves that were the decisive information source,” Raymond explained.

The Conflict Observatory measures its impact in multiple ways.

“The first and most important is — are we helping to stop the abuse and to have accountability for the abuse? Or in the case of the children’s report — have the kids come home?” Raymond said. 

In the majority of cases he’s tracked in his career, victims of incidents he helps record are typically already dead. So, using open-source technologies to save people in near-real-time feels to him new and rare.

“But the number one impact for us is — did the people come back? Or did the grave get found? Did the perpetrator go to trial? Did the abuse stop?” Raymond said. 

To date, numbers available to his team from civil society groups facilitating repatriation suggest that approximately 400 children have been returned, he confirmed on Friday.

His crew also considers whether and how their work affects U.S. and global policies. 

“I would say yes, we have, on Ukraine. We’ve put issues on the table — from destruction of grain silos to the children’s issue to the filtration camps — in a way that they were not on the front burner until we really were able to show the meta-trend impact of incidents in aggregate and summarize that and say, ‘Here’s the so-what,’” Raymond said.

Another level of impact, which he noted is easiest to quantify, is if the reports inform the public and change the narrative and literacy around issues being documented.

“Ukraine has the richest production of open-source information in a conflict that I’ve ever seen,” Raymond also told DefenseScoop.

He noted that his team has been able to apply open-source data to reach massive investigative conclusions, where the imagery is not leading — but corroborates — phenomena. One reason why Raymond finds that notable is because future conflicts “are going to look more and more like Ukraine.” 

“And they’re going to look like [that] when we’re doing human security work and evidence collection work. We’re doing that in a highly kinetic, social, digital terrain where, basically, we are changing the environment as we are observing the environment,” he said.

The ‘social digital terrain’

When asked about the original roots of the Conflict Observatory, Raymond said: “Well, the story really begins in Biloxi, Mississippi, in 2005.”

At the time, he was a humanitarian aid worker responding to Hurricane Katrina with Oxfam America. Sitting in old church pews there, surrounded by rubble, he almost immediately noticed that Black communities went “days” in the initial response after hurricane landfall without receiving government assistance.

“And then we went to the wealthier, whiter communities — and they had aid pretty immediately,” Raymond said. 

As he tried to understand why the response in some communities was so slow, he observed that “the population in African American communities had less cell phones than the communities uptown.” He eventually defined that phenomena as “digital invisibility” to highlight this disparity in how quickly responders were getting reports of need and seeing the damage.

Raymond went on to write a memo for Google about how its commercial data and satellite imagery could have improved the common operating picture and decision loop on identifying those demographic groups that were being left out of the response. However, back then access to cell phones was much more of a luxury than it is nowadays. “So, that memo sat in a drawer until October of 2010,” Raymond said. 

Around that time, he was working on an entirely different project investigating abuses against detainees in Defense Department and CIA custody for Physicians for Human Rights, when he got a call out of the blue from someone with a request.

“He was in a hotel room, right then, eating a margherita pizza with movie star George Clooney, and he wanted to share the Google memo with George Clooney, because George Clooney was looking to build a civilian reconnaissance capability to detect potential attacks against civilian communities on the border between Sudan and what was about to be the newest country in the world — South Sudan — which was seceding at the start of 2011, from Sudan under the Nova Scotia accords,” Raymond said.

That conversation eventually led to a check from Clooney, Brad Pitt, Don Cheadle and Matt Damon to build what became the Satellite Sentinel Project. In collaboration with the Harvard Humanitarian Initiative, the project essentially encompassed attempting to be “the first use of commercial satellite imagery — which had once been really only the domain of ‘Fortune 500’ companies, the U.S. intelligence community and the defense sector,” Raymond noted — for human security analysis and civilian protection in Sudan.

He said the roots of the open-source analysis and action-based intelligence (ABI) his team at the Conflict Observatory are doing now “at scale and at tempo,” trace back to that work.

During the first and second battles of Fallujah, the Marines used ABI to fuse the temporal and spatial signatures, or the time and place metadata, across many multi-source intelligence streams. They started building spreadsheets to generate means of observing what experts refer to as “the mosaic effect” — the linking of multiple datasets to reveal new information — on what was happening there, for purposes of fighting that battle.

“We basically took the ABI concept and started to use it with commercial satellite imagery for detecting and documenting attacks on civilians,” Raymond said. “What we’re doing now is just an expansion of that proof of concept.”

Now, the Sudan Conflict Observatory remote monitoring platform documents conflict activity in that country, and tracks damage to infrastructure caused by combatants, military equipment movements, and rapid population shifts. Results of those reports are often shared by the State Department.

Offering metaphors on some of the differences between the team’s work in the different locations, Raymond said: “In Ukraine, we’re ‘CSI’ or we’re ‘Law and Order’. We’re Jerry Orbach and Chris Noth trying to collect the shell casings in digital plastic bags for prosecutors. And then in the case of Sudan, we’re the alarm system, we are basically dispatched for 911, and we’re trying to tell humanitarian actors and protection actors and policy actors where they should be focused and what’s next.”

Those demonstrate two inherently different missions. 

“But what joins them together is accountability, early warnings, early response, civilian protection. What joins them together is that the means and the methods are the same — and they are happening in a place which is extremely contested,” Raymond told DefenseScoop.

Technology and societies have changed a lot since 2005, and so has what he deems “the social digital terrain,” or the intersection of informatics, social systems and physical infrastructures.

Access to such infrastructure — and contestation and control of pipes, cell towers and fiber optic networks for strategic advantage between state and non-state actors — are now a feature of modern-day armed conflicts, in Raymond’s view. Civilian security and human rights reporting and protection are thus “hanging in the balance of that infrastructure space that is being contested by companies, by states and by non-state actors — who all understand that controlling that space is not an accessory to the battlefield, it is the battlefield terrain,” he said.

The level of OSINT now occurring in Ukraine isn’t comparable to Sudan, in some ways because, in Africa, the digital terrain is typically more fragile. 

“The Ukraine reports are strategic for prosecutorial direction, and for strategic view of what’s happening and broad trends of potential criminality by Russia’s forces. In the case of Sudan, it’s more on the tactical level of tactically what are forces doing that affect the humanitarian delivery space, the civilian protection space and the day-to-day policy space,” Raymond said. 

He also praised the Biden administration for deciding to authorize the Conflict Observatory and other open-source projects for the public before Russia invaded Ukraine.

“It represents a major shift, which is that, in the past, the goal of decision support — what we produce — happened in a classified space for decision makers,” who Raymond noted would choose methods and a course of action, “then maybe tell the American people” about their approach.

“Now, this is the shift [toward] what happens when you take these capabilities and start to talk to the world with them and put them in a way where they can be read by anyone. What does that do for countering disinformation? What does that do for holding people accountable, both diplomatically and legally in court? What does that do for changing the behavior of bad actors and speeding the decision calculus of good actors? I don’t have all the answers to that — but now we can ask those questions, because we have the Conflict Observatory and other similar activities,” he said.

Raymond also hinted that his team ultimately aims to expand initiatives to nations beyond Ukraine and Sudan down the line. 

“Well, we did one, and now we do two. If we don’t mess up, maybe we’ll do three. But at this point, the genie cannot be put back in the bottle. Whether there is a different party in power after the next election — whether this program is renewed or not — the proof of concept has happened. The way I think about it is like [legendary U.S. military pilot] Chuck Yeager with the Bell X-1 Glamorous Glennis. I don’t know what planes are going to come next, but they’re all going to be built in reference to the fact that we broke the sound barrier,” Raymond said.