Bi-Annual Newsletter: Second Edition

Newsletter Highlights

1. Overview

2. ROXANNE’s clustering with other security projects 

3. An ethical approach to building the ROXANNE platform

4. ROXANNE platform: Training and 1st field test implementation

5. Speaker recognition and clustering on NIST SRE data

6. End-user requirements and use cases for a strategic approach of the ROXANNE  platform

7. ROXANNE meets K5: Exchanging experiences, state of the art and research opportunities in Data analysis methods and tools in criminology 

8. Blogs and articles

9. Publications

10. Events

11. Project at a glance

12. Consortium

Overview

We have also defined an interesting use-case scenario, which is currently analysed by the consortium partners and is planned to be demonstrated to project internal partners and other stakeholders during the field test event (planned for September 30, 2020). The data is collected by the consortium following the real-world scenarios in association with our consortium partner - Police of Czech Republic .
 
Finally, in order to start planning the future exploitation of ROXANNE 's technical results, whether collective or individual, we have performed two activities: (i) created an inventory of all components provided by partners or that will be developed during the project, and (ii) designed a questionnaire to support the elaboration of the exploitation plan.

ROXANNE’s clustering with other security projects

An ethical approach to building the ROXANNE platform

ROXANNE Platform: Training and the first field test

Successful field testing contributes towards the true, unbiased perception and adoption of the technologies and solutions proposed, thus assisting the developing team to understand and fix any potential issues, in order to present end-users with a final product which is desirable and ready to use. 

To  ensure the development of the ROXANNE platform over 3 years  and continuously collect and apply feedback from LEAs, the ROXANNE Field Test Guidance Methodology (FTGM), which is based on the Trial and Guidance Methodology principles (TGM), has planned 3 field tests aligned with 3 operational use-cases with increased complexities represented as follows:

• Field test 1: Speech and Text Analytics with preliminary Network Analysis
• Field test 2: Reduced complexity investigation use-case and first full demonstration of the ROXANNE platform
• Field test 3: Full complexity investigation use-case and second full demonstration of the ROXANNE platform 

The 1st field test will take place on 30th September, where a wide group of end users, namely Law Enforcement Agencies (LEAs), Stakeholder Board members, Ethics Board members as well as DG Home experts,  will provide input and feedback on the platform including, where appropriate participation in its testing and evaluation under realistic conditions. The initial planning was for the field test to be held physically in KEMEA’s premises. However, due to the COVID-19 outbreak, we have been obliged to re-design the field test activity, and organise it online as our top priority is the health of attendees. This first field testing targets the successful integration of all available components in the ROXANNE platform with the active participation of LEAs (internal and external).

During these tests, training activities as well as evaluation sessions will take place at the end of each field test respectively, to provide a well-rounded assessment for further analysis and development. For the purposes of training and use of the ROXANNE system, several trainees coming from different backgrounds such as LEAs, researchers, technical partners and other relevant stakeholders, an online educational platform will be used. In particular, the learning platform (training module) of ROXANNE project is based on Moodle, which is widely used by several universities and other educational organisations worldwide. Due to the web-based nature of the platform, synchronous and asynchronous modes of training can be supported. In the synchronous mode the communication between tutor and learner is simultaneous, while in the asynchronous mode the trainee is able to interact at any time, without the trainer’s presence.

Speaker recognition and clustering on NIST SRE data

Speaker recognition is one of the important technologies of the ROXANNE platform. Typically four tasks are studied in speaker recognition research namely: "verification" -- are two speakers the same?,  "identification"-- who is the speaker?,  "diarization" -- who spoke when? and "clustering". Speaker clustering aims to group a set of utterances so that recordings from the same speaker are in the same group, a.k.a. cluster. By adding constraints to the clustering process, it can be reduced to any of the other speaker recognition tasks. For example, if we have four utterances and assume that three specific utterances are in the same cluster, it remains for the clustering process to say whether the fourth utterance should be grouped together with the other three or be in a cluster of its own. This is equivalent to speaker verification where the three specific utterances are the enrollment utterances and the fourth utterance is the utterance to be verified. The ROXANNE platform should be able to cover more general scenarios. For example, there might be a set of utterances for which the speaker identity is known. The clustering process should then be constrained to maintain the known speaker identities while utterances for which the speaker is not known can be grouped either together with utterances with known speaker identity or form a new cluster. In addition, natural constraints, for example that utterances coming from two sides of a call cannot be by the same speaker, should be sustained.
 
An important aspect of the ROXANNE platform is the integration of various technologies, e.g., speaker recognition and network analysis. For research and development (R&D), we therefore need databases that are suitable for experiments on both tasks. Unfortunately, there are not many such databases available. One of the possibilities we have investigated is to use corpora collected by the National Institute of Standards and Statistics (NIST) which are intended for speaker recognition evaluations and select data from them which is suitable also for network analysis. To this end we searched for "speaker networks” across several NIST databases such that every two speakers, A and B, in a network are connected to each other in the sense that speaker A has talked to someone, who has talked to someone,…, who has talked to speaker B. In this way we identified two speaker networks with approximately 2000 and 500 speakers respectively. During the first year of the ROXANNE project we have developed initial technologies for speaker recognition and network analysis on this data. In the remaining two years, these technologies will be further adjusted to the needs of the ROXANNE Platform as well as combined based on probabilistic rules.

End-user requirements and use cases for a strategic approach of the ROXANNE platform

Today, LEAs face increasing challenges when they are tackling organized crime activities. In general, it is at the utmost priority for them to be able to investigate a large number of criminal cases at the same time without losing situational awareness. Considering  the fact that nowadays “intelligent devices and networks” have become  a crucial part of our daily life  each of these cases may last for several years as they involve not only hundreds of suspects but also a great range of the aforementioned devices. In some cases, suspects may speak different or even undetermined languages. They may also use special keywords and jargon. That requires skilled personnel, whose skills cannot be readily determined. Furthermore, the  collected data is increasingly heterogeneous; data can be structured (e.g., call detail records) or unstructured (e.g., audio from lawfully intercepted conversations, video from CCTV systems, text communications, codes with information that is not  easily decrypted etc.).

The high data transmission rate and the vast size of collected data demand an efficient way to store and filter this data analysis as well as to automate the most time-consuming tasks. Furthermore, the multimodal nature of the data requires a wide set of technologies for speech recognition, face detection [1] (rather than face recognition), topic detection and visual analytics; these will support investigators’ decision-making and boost their productivity while enhancing the quality of the aforementioned process. At the same time, investigation from a software perspective should build upon human expertise and critical thought, so that the investigation is completed successfully.

Rather than substituting humans, the ROXANNE project targets the cooperation and successful integration of human contribution and network analysis tools to eliminate organized crime activities in the near future. For that reason, a set of tailor made use cases will be developed in close collaboration with experts (project external advisory, stakeholders boards and INTERPOL΄s global network of law enforcement). In addition to that a worldwide survey was prepared and disseminated to LEAs in more than 194 countries,  which gathered and analysed legal, technical and operational requirements of end-users in the field of network analysis and speech analytics to design detailed use-cases based on expert input.

[1] Face detection technologies seek to detect the presence of people’s faces within digital images, while face identification goes one step further and attempt to associate a face with other personal information (e.g., name, ID, etc).

ROXANNE meets K5: Exchanging experiences, state of the art and research opportunities in data analysis methods and tools in criminology

Fighting organized crime remains an important but challenging task for Law Enforcement Agencies (LEAs) in all countries. The information and communication technologies are changing the world, driving criminology and computer science to go hand in hand to successfully withstand the challenges of this development. An example of how it can be successfully done is the K5 joint team of policemen and computer scientists from the criminal police Ulm in Germany that in their daily work are carrying out large digital data analysis for investigations and forensics purposes. 

K5 and Michael Hedderich (USAAR/ROXANNE) met at International Symposium on Text-mining for Police Analysts in Ulm (Germany). Both the teams realised that the main challenges highlighted by the K5 team, were also corresponding to the research question of ROXANNE with the aim to reduce burden through intelligent technology enhanced data analysis. The team proposed to organise a meeting which took place 20 - 21 August, 2020. The primary purpose of the meeting was to discuss the state-of-the-art data analysis methods and tools applied in criminology as well as research opportunities that are of interest for both, ROXANNE and K5.

The very friendly and open atmosphere of the meeting allowed for a deep exchange of experiences and ideas. K5 presented their organisational structure and processes to solve criminal cases. ROXANNE presented the main objectives of the project and raised some cases that we could solve better than existing approaches. 

The main challenges K5 is facing include the manual work overhead, filtering the relevant source of information from the non-relevant one in a casework, as well as spending 80% of time with data cleaning and normalization. Additionally, K5 mentioned the lack of technical solutions as there exist only a few software solutions that could be potentially used for specific tasks (e.g., analyzing geo data, filtered manually, and combined in an analyst notebook), and the specialists still need to merge the results based on their experience manually. ROXANNE has a potential to inspire the K5 team and demonstrating new technologies which include running visualisation examples of social graphs, automatically extracted from a large number of digital voice records.

Overall, we had a warm and open meeting with the Ulm police department that helped both teams to establish good cooperation. The meeting resulted in a concrete collaboration plan which will be not limited only in exchanging the ideas and experiences, but after a formal approval, could include the opportunity to evaluate the algorithms developed in ROXANNE on real data collected by K5.

Blogs & Articles

• Facial Recognition That Predicts ‘Criminality’ Sparks Bias Debate

• Face biometrics forecast to surpass $15B by 2027 as verticals and applications expand

• How Law Enforcement Agencies are Using Biometrics?

• Facial recognition retools for masked faces

• On the trail of the ‘Ndrangheta

• High-quality fingerprint enrollment still underpins government identity systems

• Robust link prediction in criminal networks: A case study of the Sicilian Mafia

• NIST Launches Studies into Masks’ Effect on Face Recognition Software

Publications

Events 

Project at a glance

ROXANNE (Real Time Network, Text, And Speaker Analytics For Combating Organized Crime  Research & Innovation Action) is a collaborative research and innovation project under the topic SU-FCT02-2018-2019-2020 (Technologies to enhance the fight against crime  and terrorism (Grant number 833635) that contributes to the fight against crime and terrorism through the development of cutting-edge technologies.
 
The project, funded by the EC with €7 million, was officially launched in September 2019 for a three-year period (end date August 2022) and  is coordinated by the IDIAP Research Institute. The multidisciplinary consortium with extensive practical experience in crime investigation and monitoring, as well as with excellent research and development abilities, consists of 24 partners, including INTERPOL and 10 LEAs, SMEs, industrial high-tech companies and academic institutes from 16 countries.

Consortium