Assess trained brains with managed simulators


  • Total time to complete: 15 minutes
  • Active time: 10 minutes

Use custom assessment functionality in the Bonsai UI and CLI to run user-defined experiments that evaluate how well your Bonsai brain is performing in specific scenarios.

Tip

If you want to fully customize your assessment episode configurations, you can create your own assessment configuration file and upload it instead.

Before you start

  • You must have a simulator. You cannot train or assess a brain without a simulator. Your simulator can be local (unmanaged) or packaged in Bonsai (managed).
  • You must have a trained (or partially trained) brain. You can run custom assessments on your AI at ay time once training begins, even if the brain is still in the process of training.

Start an assessment with the Bonsai UI

  1. Open the Bonsai UI.
  2. Select the brain version you want to assess.
  3. Select on the Train tab.
  4. Select the + New assessment button in the assessment panel.
  5. Set the number of episodes for your assessment.
  6. Specify a value, set of values, or value range for any configurable variables you want to constrain using Inkling syntax.
  7. Click Generate.
  8. Review the generated JSON assessment episode configurations. Bonsai runs one assessment episode for each configuration listed in the file. Bonsai uses simulator defaults for any value or episode configuration you leave blank.

Assessment configuration modal

assessment.

When the JSON file looks correct, click Run assessment to start the assessment. Assessment automatically starts with the simulator package defined in your Inkling file. For example, if your simulator statement looks like the following code snippet, your assessment automatically starts with the Moab simulator package:

source simulator MoabSim(Action: SimAction, Config: SimConfig): ObservableState {
  package "Moab"
}

Tip

Bonsai pulls the list of assessment configuration variables from your Inkling file. To include additional configuration variables, make sure to update the relevant section of your Inkling file.

Start an assessment with the Bonsai CLI

  1. Create an assessment configuration file.
  2. Use the brain version assessment start command to start the assessment:
    bonsai brain version assessment start             \
      --brain-name=BRAIN_NAME                         \
      --file=PATH_TO_ASSESSMENT_CONFIG_FILE           \
      --concept-name=CONCEPT_NAME                     \
      --simulator-package-name=SIMULATOR_PACKAGE_NAME \
      –-instance-count INSTANCE_COUNT
    

Logging happens automatically for managed simulators, so you do not need to make a separate logging command.

Evaluate assessment results

After your assessment completes, you can review the high level results by clicking on the relevant assessment in the Bonsai UI.

Data from custom assessments is stored in the Log Analytics Workspace for your Bonsai workspace. You can query assessment data in two ways:

  • Follow the Querying logs guide to query data using KQL.
  • Use the open source assessment data Jupyter notebook to import your data into Jupyter and analyze it there. The instructions included in custom-assessment-logs.ipnyb will walk you through the process.

For more detailed instruction on interpreting assessment data, follow the Evaluate assessment data tutorial.

A practical example

The following example walks you through the process of investigating how Moab performs with different plate heights using the Bonsai UI assessment feature.

Given that Moab plate height is configurable on the physical device, the goal of the example assessment is to determine the heighest plate height at which the device can still balance the ball well.

  1. Open the Bonsai UI.
  2. Select the Moab sample on the welcome page.
  3. Select the new Moab brain from the brain list.
  4. Select the Train tab.
  5. Select Train to start training the demo.
  6. Once the simulators have started, click + New assessment.
  7. Review the configuration variable list in the assessment modal. Notice that plate height is not on the list. Height is not listed as a configuration variable because it is not included by default in the Moab Inkling file.
  8. Select the Teach tab to open the Moab Inkling file.
  9. Scroll to the simulator statement:
    source simulator MoabSim(Action: SimAction, Config: SimConfig): ObservableState {
    
    Configuration variables are defined in the SimConfig type used by the simulator statement.
  10. Scroll to the SimConfig definition in the Inkling.
  11. Add height_z as a configuration variable and constrain the value to be a float between 0.1m and 0.2m.
    type SimConfig {
        # Model initial ball conditions
        initial_x: number<-RadiusOfPlate .. RadiusOfPlate>, # in (m)
        initial_y: number<-RadiusOfPlate .. RadiusOfPlate>,
    
        # Model initial ball velocity conditions
        initial_vel_x: number<-MaxVelocity .. MaxVelocity>, # in (m/s)
        initial_vel_y: number<-MaxVelocity .. MaxVelocity>,
    
        # Range -1 to 1 is a scaled value that represents
        # the full plate rotation range supported by the hardware.
        initial_pitch: number<-1 .. 1>,
        initial_roll: number<-1 .. 1>,
    
        # Starting height of the Moab plate in meters (m) 
        height_z: number<0.1 .. 0.2>
    }
    
  12. Select the Train tab to go back to the assessment list.
  13. Review the configuration variable list in the assessment modal. Notice that height_z now appears in the configuration variable list.
  14. Configure height_z in the assessment modal to be between 0.1m and 0.15m.
  15. Change the episode count to 100.
  16. Assign constant values to all other configuration variables as indicated below.

Example assessment configuration

- initial_y: 0.5

When you select Generate to create an assessment configuration file, the corresponding JSON output contains 100 episode configurations. The values for height_z will be uniformly distributed between 0.1m and 0.15m across all 100 episodes while all other configuration variables are held constant.

When you are ready, select Run to start the example assessment.