Guidelines for Developers and QA Teams

This page complements the paper "Runtime Verification and Field Testing for ROS-Based Robotic Systems" and is an online repository of the guideline catalog for ROS-based Robotics projects.

You can further find the guidelines and information on the evaluation process. Reproduction kits, specifications, and accompanying code can be found in our replication package.

Guideline Catalog

Guidelines

Constraint identification

1. CI1

   Identify timing constraints

2. CI2

   Identify security and privacy constraints

3. CI3

   Identify safety constraints

Code design and implementation

1. CD1

   Strive for ROS nodes with single responsibility

2. CD2

   Ensure global time monotonicity of events and states

Instrumentation

1. I1

   Provide an API for querying and updating internal lifecycle

2. I2

   Provide an API for logging and filtering

3. I3

   Provide an API for injecting faults in execution scenarios

4. I4

   Isolate components for testing

Prepare Execution Environment

1. PE1

   Understand the overhead acceptance criteria

2. PE2

   Create models for runtime assessment

Specify (un)desired behavior

1. SDB1

   Define properties using a logic-based language

2. SDB2

   Use Domain Specific Languages (DSLs) to specify properties

3. SDB3

   Use languages and tools to scenario-based specification of test cases

Monitor and Test Automation

1. MTA1

   Improve the robustness of the system by performing noise and fault injection

2. MTA2

   Exploit automation for test case generation, test case prioritization and selection, oracle and monitor generation

System Execution

1. SE1

   Use record-and-replay when performing exploratory field tests

2. SE2

   No GUIs! Prioritize headless simulation

Analysis and Reporting

1. AR1

   Perform postmortem analysis to diagnose non- passing test cases

2. AR2

   Use reliable tooling in order to manage field data