Behaverse is a comprehensive software system designed to accelerate, integrate and scale-up cognitive science research and to promote best practices and open science. Behaverse is GDPR compliant, multilingual and multiplatform. It comprises six galaxies, each containing several components (planets). These galaxies are:
The Activities galaxy comprises software systems and frameworks that support interactive digital activities subjects can engage with and produce data.
Behaverse comprises video games designed to assess and improve cognitive abilities. The scientific literature has highlighted multiple hypotheses to explain cognitive training. Behaverse’s training games attempt to translate those hypotheses into concrete games and test their efficacy in improving cognitive abilities.
Behaverse comprises a large battery of cognitive tests covering a wide range of domains, including attentional, perceptual, cognitive and motor abilities. This battery consists of multiple Cognitive Assessment Engines which are highly customizable (including the instructions, tutorials and practice blocks) and can instantiate many classic cognitive tests.
The Behaverse Cognitive Assessment battery is GDPR compliant, multilingual, multiplatform and supports the collection of fine-grained behavioral data while being robust to network connectivity breaks.
Asking people questions is a major research method in behavioral sciences, be it via standardized questionnaires (e.g., a personality questionnaire) or context sensitive, targeted questions (e.g., experience sampling).
Contrary to existing off-the-shelf solutions, our in-house survey system is GDPR compliant, multilingual, supports the collection of event data (e.g., key strokes, mouse trajectory), uses a consistent data model across all use-cases and decouples data structures and backend architectures from the survey viewer (front-end). This allows us to display questions on different devices (e.g., survey in a web browser, an experience sampling prompt within a mobile video game) while maintaining coherent data structures, which greatly facilitates their processing and subsequent analyses.
Behaverse includes video games specifically designed for educational research, both for assessment and training. We released Mathemarmite on the Google Play and App Stores, a mobile video game for 3-6 year old children, designed to study and improve early mathematical (in particular counting) and cognitive abilities. This game was nominated for Best social effect / game beyond entertainment at Game Connection 2019.
The Data galaxy contains software systems, frameworks and protocols that handle everything related to data management.
There are increasing amounts of behavioral data freely available on the internet with metadata standards making it easier to find them. However, there are large inconsistencies in the way those datasets are structured and stored, in the way their variables are named and formatted and even in what is meant by various common terms. These inconsistencies make it unnecessarily hard to reuse datasets and prevent the development of effective software tools and automatization.
Behaverse defines coherent data models for both cognitive tests and surveys that apply to a large range of use-cases. In addition, Behaverse defines concepts and best practices regarding the naming and structuring of datasets.
In Behaverse, data can flow in from many sources and in many different shapes. The Telemetry Platform helps us collect high-throughput behavioral data from participants and carry it straight to the data center of the University of Luxembourg. To enforce data protection and reliability, the system intentionally avoids using third-party online services.
Moreover, our various technology-specific codebases allow us to support many devices and use-cases, including mobile games and web applications. Nested within each of these data collecting applications, a Telemetry client module mitigates potential connectivity issues and ensures that data is effectively and reliably transferred to the platform.
The Identity Server centralizes account information for the participants in our longitudinal studies. This is important to ensure their data is collected consistently and securely by our various systems. As a security token service, it also helps secure the information flow throughout our infrastructure.
Entirely hosted at the University of Luxembourg, without third-party online services, the Identity Server is a cornerstone of our data protection enforcement and GDPR compliance. For security reasons we will not communicate further on the Identity server.
Behaverse collects rich data sets from various sources as event data, supporting virtually all possible types of data. Those raw, source event data files are however harder to process and analyse. Behaverse therefore also includes data management, extraction and conversion tools to enforce quality and consistency and to facilitate their subsequent processing and analyses.
Behaverse is used to run experimental studies and collect data for research in cognitive science. We are creating a rich, well-documented, versioned dataset library to be shared with other researchers, to serve as benchmark data for computational models and, in fine, to accelerate scientific progress.
This library will be extended in various ways during the lifetime of Behaverse. All datasets will be curated to ensure a consistent data model and facilitate reuse (using the Behaverse data model) as well as to ensure data privacy of the people sharing providing that data.
The Experiments galaxy comprises a collection of tools and frameworks designed to specify, optimize and coordinate experimental studies as well as monitor and control the flow of participants across the successive stages of a study.
Game Control is a powerful set of tools that allows experimenters to flexibly control trial-based activities, be it using procedural generation methods, presets or by specifying trial parameters extensively.
Within this framework, a game experience is structured in game blocks (e.g., instructions, tutorials, practice, test), each describing a set of parameters, that can be either fixed, sampled from a distribution, or generated using a specific algorithm. Blocks can define adaptive algorithms, which update one or more parameters based on the participant’s past performance.
Survey Control is a set of tools that allow experimenters to flexibly control surveys, first by allowing them to compose questionnaires from a database of questions and response options, second by offering the possibility to quickly compose surveys from pre-existing questionnaires and finally by the use of a dedicated syntax which supports more complex conditional branching schemes (e.g., a participant will see different questions depending on their responses to past questions).
Designed with extensibility and interoperability in mind, the Study Orchestrator lets experimenters configure and oversee the flow of participants across a sequence of activities within their study. Each step of the sequence can lead to branching a user off into another path, depending on the study requirements. Branching can be fine-tuned to allow a wide range of behaviour, from load balancing coverage across groups, to custom assignments of participants to experimental groups.
The Study Orchestrator also offers a testing website that people can log into to participate in a study. It handles user authentication, resuming participation in the study, and provides experimenters with a dashboard summarizing the progress of the study.
With a seamless integration with the online participant recruitment platform Prolific, the Study Orchestrator provides an all-in-one solution for quickly creating and publishing complex, adaptive online experiments.
Experiment Optimization contains a collection of tools to help create better experiments. In particular, this collection contains sequence generators which allow us to produce sequences of stimuli with specific statistical properties (Ansarinia et al., 2019).
This collection will furthermore include tools to optimize the assignment of participants to particular conditions of a study, or for match-making participants, to give just a few examples.
Analytics contains tools and frameworks for the processing, analysis and visualization of the data collected within Behaverse.
Currently a prototype implemented in R/Python, the data analysis pipeline will run on the High Performance Computing clusters of the University of Luxembourg. Analytics will support automated data analyses using continuous integration tools, meaning that analyses will be triggered each time the data or the data analysis code is updated.
In addition to scalability (i.e., the resource efficient analysis of large datasets), Analytics aims to increase the quality of the analysis of our data, first by defining and implementing standardized, transparent and reproducible procedures, second by using state-of-the-art data analysis and machine learning methods (e.g., PyTorch) and finally by offering the possibility to automatically reanalyse past data as models are updated.
Behaverse is a research platform to run experimental studies in cognitive sciences and to collect large and rich data sets for research. To facilitate the use of Behaverse as a research platform we will create tools (e.g., dashboards) to facilitate the exploration of the different components of Behaverse (e.g., data sets, questionnaire libraries, participant statistics).
Insights will comprise a real-time view of the data, the data analysis code, the status of ongoing analyses as well as various outputs generated by the analyses (e.g., reports). Insights will also provide access to interactive notebooks to support tinkering and exploratory data analyses.
Models contains both a library of models as well as tools to automatically run those models on the data collected within Behaverse (model fitting and comparison) and/or have these models interact with Activities to mimic human behavior.
Robots in Behaverse are artificial agents capable of interacting with the Behaverse Activities (e.g, the training game, surveys) and which may mimic human behavior. Prototypes of dummy robots (acting randomly) have been implemented to generate data for subsequent data quality and system integrity checks. In the future we will interface with state-of-the-art RL agents libraries (e.g., stable-baselines), as comparing the performance of those robots with humans may provide invaluable insights into how the human mind works.
Currently at the concept stage, Computational models of cognition aims to provide a framework and software tools to support integrative and cumulative efforts in modeling cognition.
It will contain a library of models implemented in various programming languages but following the same API. These models will be available on Docker and Singularity so they can run on the High Performance Computing clusters of the University of Luxembourg.
The models will be tested and compared using common metrics and datasets, which will be collected within Behaverse. Following the principles of continuous integration, each time new datasets are collected, all applicable models will be tested on those new datasets. Similarly as new models are introduced, they will be tested on all available relevant datasets and compared to the other models in the library.
This setting allows modelers to develop their models in their tools of choice, to more objectively and easily test the performance and scope of their models, while offering the possibility to share their models with the community (either with or without access to the source code).
Further tools will be needed for example, to explore results (e.g., dashboards) and tinker with the various models (e.g., notebooks). We believe this type of system is necessary both to promote modeling and to get a clearer sense of where we currently stand as a field.
The ultimate goal of Behaverse is to construct a “Psych Engine” (in analogy to the physics engine in video games) which will integrate all of our knowledge of the human cognitive system. The Psych Engine should be able to use a person’s historic data to simulate that person’s cognitive system and make non-trivial predictions about how that specific person would perform in various cognitive tests or games. In other words, the Psych Engine would instantiate a “digital twin” of a person’s cognitive system.
In addition to the scientific value of creating such a system, the Psych Engine may be used both to guide experiments to further our understanding of cognition (e.g., by pointing out predictions with high uncertainty), suggest interventions to help people improve their cognitive abilities and for diagnostics (e.g., unexpected performance drops in specific cognitive abilities may be indicative of a specific neurological issue).
Tools is a collection of software systems and tools that are used transversally within Behaverse, for example for monitoring or quality assurance.
Overwatch is a set of system monitoring tools that allow us to verify in real-time that the various systems within Behaverse are working properly. For example, if a particular server experiences issues, notifications will be sent to the system administrator and will be displayed on the dashboard so that immediate actions can be taken to solve the issue at hand and possibly prevent losing data.
Quality check comprises a variety of tools aimed to ensure the quality of various components of Behaverse. Thes tools include, for example, unit tests of code and data, but also code to check the readability of the instructions or the robustness of cognitive test engines.
Utilities comprises a large collection of smaller, diverse tools that are used throughout the Behaverse software systems. This includes, for example, deployment tools that allow us to quickly set up data collection servers, tools for reporting error messages when a game crashes, code to handle localization, and tools for the encryption and decryption of data, to name just a few. In general, these tools are designed to facilitate different aspects of the development and to automate as many steps as possible so that the overall system is more robust, reproducible and reusable.