Bi-Annual Newsletter: Second Edition

Edition: September 2020

Newsletter Highlights

  1. Overview

  2. ROXANNE’s clustering with other security projects 

  3. An ethical approach to building the ROXANNE platform

  4. ROXANNE platform: Training and 1st field test implementation

  5. Speaker recognition and clustering on NIST SRE data

  6. End-user requirements and use cases for a strategic approach of the ROXANNE  platform

  7. ROXANNE meets K5: Exchanging experiences, state of the art and research opportunities in Data analysis methods and tools in criminology 

  8. Blogs and articles

  9. Publications

  10. Events

  11. Project at a glance

  12. Consortium


Project testimonial by Netherlands Forensic Institute

Several European countries are joining forces in the fight against organized crime within the ROXANNE project, which started in September 2019 and will run until 2022. The main goal of this project is to combine advanced video, speech and language technologies together with network analysis into one end product, underpinned by a strong legal framework, to improve and assist police investigations. Partners from academia, industry, and law enforcement contribute knowledge and technologies. Uniqueness to this project lies in the fact that existing and new technologies are combined to map criminal networks more accurately and to achieve stronger conclusions in criminal investigations against transnational organized crime.

As NFI, we are eager to contribute to the application and improvement of state-of-the-art technologies in the forensic domain. Furthermore, we look forward to the innovative techniques ROXANNE will further develop to identify criminals and their organizations based on communication traces. Insight into relationships between people or the relationships between devices is very important in this respect. 

Ultimately, the tool is designed for the Law Enforcement Agencies (LEAs); therefore, it is important that their concerns and requirements as end users are well represented in the end product. Towards this goal, in collaboration with INTERPOL, the Greek Center for Security Studies KEMEA, and other project partners, we have gathered and analyzed valuable input from the LEAs. A year after the start of this project, all participants are preparing for the first field test, which is discussed in more detail later in this newsletter. During this test, the LEAs will have the opportunity to experience the first version of the ROXANNE Platform and provide their feedback. We are very excited to participate in this demonstration and to see all the work come together.

Progress in the last 6 months

As for many situations around the world, the COVID-19 pandemic has impacted the ROXANNE project and our partners have had to adapt their ways of working to this new situation. Some modifications to the initial planning phase were made and we had to postpone the organisation of the first field test event, originally intended to be held in person with the LEA representatives. 
In order to move forward, the consortium has leveraged various datasets including research datasets such as CSI and NIST Speaker Recognition Evaluation Collection and also collected its own simulated dataset, the ‘ROXSD’. The consortium has started to integrate the capabilities and the technical solutions which will be demonstrated during the field test set-up by KEMEA. These include speech processing from Brno University of Technology, Phonexia and SAIL LABS, text processing from IDIAP and Saarland University, video processing from Airbus, and network analysis from the University of Hannover. 
From an architectural point of view, each technical partner develops its own micro-services, each service being encapsulated as a Docker container. These services can then be orchestrated using Kubernetes and combined using Apache Nifi processing chains. AEGIS provides to the user data visualisation capabilities relying on their forensic visualisation toolkit. The ROXANNE platform is now able to run a complete processing chain on the CSI, NIST and ROXSD datasets and demonstrate their results to its users.

We have also defined an interesting use-case scenario, which is currently analysed by the consortium partners and is planned to be demonstrated to project internal partners and other stakeholders during the field test event (planned for September 30, 2020). The data is collected by the consortium following the real-world scenarios in association with our consortium partner - Police of Czech Republic .
Finally, in order to start planning the future exploitation of ROXANNE 's technical results, whether collective or individual, we have performed two activities: (i) created an inventory of all components provided by partners or that will be developed during the project, and (ii) designed a questionnaire to support the elaboration of the exploitation plan.

ROXANNE’s clustering with other security projects

As part of an initiative led by Trilateral Research, ROXANNE has joined a cluster of security projects along with CC-DRIVER, COPKIT, DARLENE, INSPECTr, PREVISION, PROTAX, and RAYUELA. The coordinators of these projects are collaborating on shared issues and topics of interest. This allows the projects to help each other in achieving greater impact and knowledge dissemination both within the projects themselves and through their networks. As joint activities progress, we intend to share news from other projects through this newsletter.
This cluster also intends that the project partners will invite each other to their events in order to share ideas and results with a look toward future research collaboration. These collaborations will enable the projects to work together and collect different perspectives on tackling common issues in order to collaborate on joint solutions. This collaboration could span research on technical, ethical, legal, and data protection issues, amongst others.
More information about this cluster of research projects, and joint activities can be found here:

An ethical approach to building the ROXANNE platform

For the ROXANNE project, compliance with ethical and legal standards are cornerstones that are essential to the success of the project and for creating a platform that could, potentially, be used by LEAs in the future. To this end, the project has implemented an ethical approach from the beginning to ensure that all of the actions taken toward delivering the project do not violate any common ethical principles. A large part of this has been analysing impacts of the technologies that will form the ROXANNE platform, and the process used to build them.

Potential impacts are considered across the life-span of the project and potential uses of the platform. Issues are considered from ethics, societal values, fundamental rights, and applicable legislation. Possible impacts that the ROXANNE project and platform could have in each of these areas are highlighted and potential mitigation strategies are suggested in order to avoid any negative impacts. These assessments are ongoing, and, once completed, should provide guidance for technical partners to produce the platform in a way that avoids negative impacts. Additionally, guidance for potential uses of the platform that do not violate ethical and legal standards will also be included.

ROXANNE Platform: Training and the first field test

Successful field testing contributes towards the true, unbiased perception and adoption of the technologies and solutions proposed, thus assisting the developing team to understand and fix any potential issues, in order to present end-users with a final product which is desirable and ready to use. 

To  ensure the development of the ROXANNE platform over 3 years  and continuously collect and apply feedback from LEAs, the ROXANNE Field Test Guidance Methodology (FTGM), which is based on the Trial and Guidance Methodology principles (TGM), has planned 3 field tests aligned with 3 operational use-cases with increased complexities represented as follows:

  • Field test 1: Speech and Text Analytics with preliminary Network Analysis
  • Field test 2: Reduced complexity investigation use-case and first full demonstration of the ROXANNE platform
  • Field test 3: Full complexity investigation use-case and second full demonstration of the ROXANNE platform 

The 1st field test will take place on 30th September, where a wide group of end users, namely Law Enforcement Agencies (LEAs), Stakeholder Board members, Ethics Board members as well as DG Home experts,  will provide input and feedback on the platform including, where appropriate participation in its testing and evaluation under realistic conditions. The initial planning was for the field test to be held physically in KEMEA’s premises. However, due to the COVID-19 outbreak, we have been obliged to re-design the field test activity, and organise it online as our top priority is the health of attendees. This first field testing targets the successful integration of all available components in the ROXANNE platform with the active participation of LEAs (internal and external).

During these tests, training activities as well as evaluation sessions will take place at the end of each field test respectively, to provide a well-rounded assessment for further analysis and development. For the purposes of training and use of the ROXANNE system, several trainees coming from different backgrounds such as LEAs, researchers, technical partners and other relevant stakeholders, an online educational platform will be used. In particular, the learning platform (training module) of ROXANNE project is based on Moodle, which is widely used by several universities and other educational organisations worldwide. Due to the web-based nature of the platform, synchronous and asynchronous modes of training can be supported. In the synchronous mode the communication between tutor and learner is simultaneous, while in the asynchronous mode the trainee is able to interact at any time, without the trainer’s presence.

Speaker recognition and clustering on NIST SRE data

Speaker recognition is one of the important technologies of the ROXANNE platform. Typically four tasks are studied in speaker recognition research namely: "verification" -- are two speakers the same?,  "identification"-- who is the speaker?,  "diarization" -- who spoke when? and "clustering". Speaker clustering aims to group a set of utterances so that recordings from the same speaker are in the same group, a.k.a. cluster. By adding constraints to the clustering process, it can be reduced to any of the other speaker recognition tasks. For example, if we have four utterances and assume that three specific utterances are in the same cluster, it remains for the clustering process to say whether the fourth utterance should be grouped together with the other three or be in a cluster of its own. This is equivalent to speaker verification where the three specific utterances are the enrollment utterances and the fourth utterance is the utterance to be verified. The ROXANNE platform should be able to cover more general scenarios. For example, there might be a set of utterances for which the speaker identity is known. The clustering process should then be constrained to maintain the known speaker identities while utterances for which the speaker is not known can be grouped either together with utterances with known speaker identity or form a new cluster. In addition, natural constraints, for example that utterances coming from two sides of a call cannot be by the same speaker, should be sustained.
An important aspect of the ROXANNE platform is the integration of various technologies, e.g., speaker recognition and network analysis. For research and development (R&D), we therefore need databases that are suitable for experiments on both tasks. Unfortunately, there are not many such databases available. One of the possibilities we have investigated is to use corpora collected by the National Institute of Standards and Statistics (NIST) which are intended for speaker recognition evaluations and select data from them which is suitable also for network analysis. To this end we searched for "speaker networks” across several NIST databases such that every two speakers, A and B, in a network are connected to each other in the sense that speaker A has talked to someone, who has talked to someone,…, who has talked to speaker B. In this way we identified two speaker networks with approximately 2000 and 500 speakers respectively. During the first year of the ROXANNE project we have developed initial technologies for speaker recognition and network analysis on this data. In the remaining two years, these technologies will be further adjusted to the needs of the ROXANNE Platform as well as combined based on probabilistic rules.

End-user requirements and use cases for a strategic approach of the ROXANNE platform

Today, LEAs face increasing challenges when they are tackling organized crime activities. In general, it is at the utmost priority for them to be able to investigate a large number of criminal cases at the same time without losing situational awareness. Considering  the fact that nowadays “intelligent devices and networks” have become  a crucial part of our daily life  each of these cases may last for several years as they involve not only hundreds of suspects but also a great range of the aforementioned devices. In some cases, suspects may speak different or even undetermined languages. They may also use special keywords and jargon. That requires skilled personnel, whose skills cannot be readily determined. Furthermore, the  collected data is increasingly heterogeneous; data can be structured (e.g., call detail records) or unstructured (e.g., audio from lawfully intercepted conversations, video from CCTV systems, text communications, codes with information that is not  easily decrypted etc.).

The high data transmission rate and the vast size of collected data demand an efficient way to store and filter this data analysis as well as to automate the most time-consuming tasks. Furthermore, the multimodal nature of the data requires a wide set of technologies for speech recognition, face detection [1] (rather than face recognition), topic detection and visual analytics; these will support investigators’ decision-making and boost their productivity while enhancing the quality of the aforementioned process. At the same time, investigation from a software perspective should build upon human expertise and critical thought, so that the investigation is completed successfully.

Rather than substituting humans, the ROXANNE project targets the cooperation and successful integration of human contribution and network analysis tools to eliminate organized crime activities in the near future. For that reason, a set of tailor made use cases will be developed in close collaboration with experts (project external advisory, stakeholders boards and INTERPOL΄s global network of law enforcement). In addition to that a worldwide survey was prepared and disseminated to LEAs in more than 194 countries,  which gathered and analysed legal, technical and operational requirements of end-users in the field of network analysis and speech analytics to design detailed use-cases based on expert input.

[1] Face detection technologies seek to detect the presence of people’s faces within digital images, while face identification goes one step further and attempt to associate a face with other personal information (e.g., name, ID, etc).

ROXANNE meets K5: Exchanging experiences, state of the art and research opportunities in data analysis methods and tools in criminology

Fighting organized crime remains an important but challenging task for Law Enforcement Agencies (LEAs) in all countries. The information and communication technologies are changing the world, driving criminology and computer science to go hand in hand to successfully withstand the challenges of this development. An example of how it can be successfully done is the K5 joint team of policemen and computer scientists from the criminal police Ulm in Germany that in their daily work are carrying out large digital data analysis for investigations and forensics purposes. 

K5 and Michael Hedderich (USAAR/ROXANNE) met at International Symposium on Text-mining for Police Analysts in Ulm (Germany). Both the teams realised that the main challenges highlighted by the K5 team, were also corresponding to the research question of ROXANNE with the aim to reduce burden through intelligent technology enhanced data analysis. The team proposed to organise a meeting which took place 20 - 21 August, 2020. The primary purpose of the meeting was to discuss the state-of-the-art data analysis methods and tools applied in criminology as well as research opportunities that are of interest for both, ROXANNE and K5.

The very friendly and open atmosphere of the meeting allowed for a deep exchange of experiences and ideas. K5 presented their organisational structure and processes to solve criminal cases. ROXANNE presented the main objectives of the project and raised some cases that we could solve better than existing approaches. 

The main challenges K5 is facing include the manual work overhead, filtering the relevant source of information from the non-relevant one in a casework, as well as spending 80% of time with data cleaning and normalization. Additionally, K5 mentioned the lack of technical solutions as there exist only a few software solutions that could be potentially used for specific tasks (e.g., analyzing geo data, filtered manually, and combined in an analyst notebook), and the specialists still need to merge the results based on their experience manually. ROXANNE has a potential to inspire the K5 team and demonstrating new technologies which include running visualisation examples of social graphs, automatically extracted from a large number of digital voice records.

Overall, we had a warm and open meeting with the Ulm police department that helped both teams to establish good cooperation. The meeting resulted in a concrete collaboration plan which will be not limited only in exchanging the ideas and experiences, but after a formal approval, could include the opportunity to evaluate the algorithms developed in ROXANNE on real data collected by K5.

Blogs & Articles

      Related Articles:




Idiap NMT System for WAT 2019
Multi-Modal Translation Task

Workshop on Asian Translation Workshop, EMNLP2019 Hongkong

Overview of the 6th Workshop on Asian Translation

Workshop on Asian Translation Workshop, EMNLP2019 Hongkong

OdiEnCorp 2.0: Odia-English Parallel Corpus for Machine Translation

WILDRE Workshop LREC 2020 

German News Article Classification: A
 Multichannel CNN Approach

ETAEERE 2020 conference, India, March 2020

Idiap Submissions to Swiss German Language Detection Shared Task

Swiss-German Language Detection Shared Task, SwissText and KONVENS 2020 

Incremental Semi-supervised Learning for Multi-genre Speech Recognition

ICASSP 2020 conference, Barcelona, May 2020

Distant Supervision and Noisy Label 
 Learning for Low Resource 
 Named Entity Recognition:
  A Study on Hausa and Yorùbá

Accepted by the workshop PML4DC
 under ICML 2020

Idiap & UAM participation at GermEval 2020: Classification andRegression of Cognitive and Motivational Style from Text

GermEval 2020 Shared Task on the Classification and Regression of Cognitive and Motivational style from Text,
 SwissText and KONVENS 2020

Robust link prediction in criminal networks: A case study of the Sicilian Mafia

Expert Systems with Applications


Analysis of ABC Submission to NIST SRE 2019 CMN and VAST Challenge.

Proceedings of Odyssey 2020 The Speaker and Language Recognition Workshop, available in iSCA archive. 

Probabilistic embeddings for speaker diarization

Proceedings of Odyssey 2020 The Speaker and Language Recognition Workshop, available in iSCA archive. 

Utilizing VOiCES dataset for multichannel speaker verification with beamforming

Proceedings of Odyssey 2020 The Speaker and Language Recognition Workshop, available in iSCA archive. 


Consortium meeting:

The ROXANNE consortium meeting held virtually on 24-25 August 2020 and attended by the 24 partners including LEAs and SMEs from 16 countries. The two-day meeting started with a short presentation by each of the partners about their contribution to the ROXANNE project in the last 6 months and the plan for the next 6 months. Apart from the technical presentation by each of the work package leaders for their respective packages, and a project board meeting, the consortium meeting discussed the planning for the upcoming ROXANNE field test event scheduled  on 30th September 2020.

Events attended:



Interspeech 2019

15-19.09.2019 Graz, Austria

Analyst & Advisor Day at Capgemini

04.12.2019 Paris, France

Germany International Symposium on Text mining for Police Analysts 

27.11.2019 Ulm

IClR 2020 Workshop


Legal, ethical and societal challenges posed by the introduction of AI and biometric technologies in law enforcement applications


WILDRE Workshop (LREC2020)


SwissText and Konvens 2020


International Business
 Meets Technology Conference


Odyssey: The Speaker and Language Recognition Workshop

Virtual meeting (Tokyo, Japan).
 The proceedings for Odyssey are available in the ISCA Archive

CoU (Community of Users) FCT Workshop on organised crime and cybercrime

June 2020

Workshop on legal and ethical clarifications, organised by DG JUST and DG RTD EC organisations

June 2020


Upcoming Events:



First field test : Speech and Text Analytics with preliminary Network Analysis


Interspeech 2020 (Virtual)

25-29.10.2020 (Shanghai, China) 

NIST Speaker Recognition Evaluation (SRE) 2020

The challenge aims to contribute to the direction of research efforts and the calibration of  technical capabilities of text independent speaker recognition. ROXANNE plans to participate as a team in it.

Séminaire Traitement del'Information Multimodale


Nicosia Risk Forum 2020 (Online)


Project at a glance

ROXANNE (Real Time Network, Text, And Speaker Analytics For Combating Organized Crime  Research & Innovation Action) is a collaborative research and innovation project under the topic SU-FCT02-2018-2019-2020 (Technologies to enhance the fight against crime  and terrorism (Grant number 833635) that contributes to the fight against crime and terrorism through the development of cutting-edge technologies.
The project, funded by the EC with €7 million, was officially launched in September 2019 for a three-year period (end date August 2022) and  is coordinated by the IDIAP Research Institute. The multidisciplinary consortium with extensive practical experience in crime investigation and monitoring, as well as with excellent research and development abilities, consists of 24 partners, including INTERPOL and 10 LEAs, SMEs, industrial high-tech companies and academic institutes from 16 countries.

LinkedIn LinkedIn
Twitter Twitter
Website Website


This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement N° 833635.

View this email in your browser
Copyright © 2020, All rights reserved.

Our mailing address is:

This email was sent to <<Email Address>>
why did I get this?    unsubscribe from this list    update subscription preferences
The Idiap Research Institute · Rue Marconi 19 · Martigny 1920 · Switzerland

Email Marketing Powered by Mailchimp