BoxLab Visualizer

System requirements
The BoxLab Visualizer can be installed on the Microsoft Windows XP (Service Pack 2) operating system or later (tested on Windows Vista, Windows 7). At least 1GB of system RAM and a Pentium-4 or newer processor is recommended. Currently installation on Macintosh, Linux, or other platforms is not supported.

A sample version of the visualization/annotation application is available here:
BoxLabVisualizerInstaller Version 1.1.3 (500KB)

BoxLab Visualizer - Current Release

Sample Data
A sample dataset is available here. Download and unzip to a desired location on your local hard drive. You will later use the BoxLab Visualizer to open the dataset from this location. You may delete the zip archive after its contents have been extracted to the desired location. (1.7GB)

I. About
The BoxLab Visualizer is a tool for viewing and annotating BoxLab datasets. For more information, please visit

II. Getting Started
To start the Visualizer, launch the application from the Windows Start Menu under a folder called MIT.

III. Loading Data
After starting the application, you will be prompted for a dataset to view.


Click on the folder icon to bring up a folder browser dialog. Choose the folder corresponding to the data you wish to view.
  • If you want to view multiple days of data, choose the folder that is above the Day Folder(s) that you wish to view. (See example below.)
  • If you want to view one day of data, choose that Day Folder.
  • If you want to view one hour of data, choose that Hour Folder in the Day Folder it corresponds to. (Note: This will load the entire day.)


Click "OK" once you have selected the proper folder. The directory should appear in the text box.


Click on "Start Show" to begin viewing the data.

IV. Visualizer Components

a. Data Source Selection

Begin by selecting the day you wish to view from the panel in the top left.


The hours of data associated with that day appear under it. The numbers on the right of each line indicate the hour in a 24-hour format. If a number is missing, there is no data folder for that hour. You may select a "data colorizer" that will be used to reflect sensor activations or amount of activity present in each hour of data. Choose a colorizer with the buttons below the playhead control. Some of the colorizer options are illustrated below:

Object MITes
Motion MITes
The amount of activity recorded by the video sources as inferred from the variable frame rate recording software.
The presence of activity labels for a given dataset is conveyed through the use of orange markers that indicate the start time for an activity annotation.
Object movement (acceleration) sensor activation detected.
Passive infrared sensor activation
White indicates no activity.
Yellow indicates little activity.
Red or purple indicates lots of activity.
Hours with a substantial number of orange lines suggest a lot of varied activities occurred during that time.
Purple indicates sensor activation
Red indicated movement in proximity to a PIR motion sensor
The method of data collection may cause this to appear almost completely orange even when there is not continuous activity.

Annotation image goes here
Object MITes image goes here

Click on the hour you wish to view to bring up the video.

b. Video Screen and Selector

Selecting an hour will bring up all the corresponding video views. (Any unused space for views will be blue).


Double-clicking on a view will initiate a single-view layout.


You may change the view layout with the buttons in the top right corner of the video screen.


As currently implemented, when switching from a small number of video displays to a larger number, you must reassign the lost video streams to their former places (as described below). A strategy for maintaining the placement of video streams will be implemented in a future revision.

Clicking on the video screen will toggle pause and play, depending on your currently selected play speed and direction (see the "Playhead Control" section for more information). Right-clicking will bring up the annotation menu (see the "Annotation" section for more information).

At the bottom of the Visualizer are the available video and audio streams for the selected hour. Only one audio stream may be selected at a time, and currently it will only play when the video is playing forward at normal speed.

To change the placement of individual video views, hover your mouse pointer over the view you want to move. An overlay of the current view layout will appear.


Click on the space you wish to move the view to. You can also change the view layout by right-clicking on an individual video view. To apply a new view layout, click on a space in the new layout to place the current video view. (Note: Occasionally the overlay will stick to the video view after you move your mouse away. This is a known bug.)

c. Playhead Control

The three buttons on the left are used to fast-backward, play, and fast-forward the video, respectively. Right-clicking on the play button will toggle its direction between play forward and play backward.


The timeline slider, along with the Time display, indicates the current time of the video view(s).

Clicking on a button will give it color (green for forward play, orange for backward play) and change its symbol to "pause". Click on it again to pause the video.


The small numbers on the left and right indicate the speed of fast-backward and fast-forward. They can be changed individually by right-clicking on the number and selecting a speed. Clicking on the number will change its speed to the next in the list.


Drag the small circle along the slider to change the current time, or click on the slider to make the circle jump to a time. The selected colorizer scheme will appear in the background of the slider. Right-click on the slider to magnify a section of the timeline (in the large circle below) for more fine-grained time selection.


The volume slider can be used to adjust the volume of the audio.

d. Annotation Management

Annotation management is on the right side of the Visualizer.


  • Right-clicking on an annotation will bring up an annotation menu (see the "Annotation" section for more information).
  • Double-clicking on an annotation tab will jump the video time to the beginning of the annotation, if that start time is within the current hour. If it’s not within the current hour, it will change the current video time to the start or end of the current hour, depending on which is closer to the start time of the annotation. Subsequent double-clicking on an annotation tab will jump the video time between the beginning and end of that annotation. The current time is highlighted in white
  • [CURRENTLY DISABLED] Pressing CTRL + Z (undo) from anywhere in the application will cause the last change made to an annotation to be reversed.

Click on the small yellow rectangle on the right side of any annotation to toggle between expanded and collapsed. Holding the CTRL key and expanding or collapsing an annotation will cause all annotations to assume that state. [CURRENTLY BROKEN]

The expanded version lists the certainty ratings of the annotation next to the times they begin.


The annotations shown in the topmost portion of the annotator interface are those including at least one time stamp with a stop time (i.e. "Not happening"). They are listed under the time they begin, in chronological order. Only the current annotations with a stop time in the current hour will appear in the list. For annotations to appear in the list, they must belong to a currently selected annotation set.

The color of the annotation is determined by the Annotation Set selection interface. Right-clicking on one of the annotation sets will allow you to change the color used for the annotations associated with it. During editing and playback, the annotations will reflect their state using variations on their base color:
  • A lighter shade of the annotation color indicates that the label is currently happening.
  • A darker shade of the annotation color indicates that the label is not currently happening.
  • A red color indicates that the current time of the video currently at the start time of the annotation.
  • A gray color indicates that the annotation has a logical error (such as multiple "Not happening" or two of the same rating in a row).


Partial annotations are those without a stop time. They are listed in chronological order based on their start time. Only the partial annotations from the current hour or previous hours will appear in the list.
  • A red color indicates that the current time of the video is the start time of the annotation.
  • A light orange color (see the first annotation below) indicates that the annotation is currently happening (its start time is before the current time of the video).
  • A dark orange color (see the last two below) indicates that the annotation is not currently happening (its start time is after the current time of the video).


e. Keyboard Shortcuts

Step Forward/Backward a Frame
Set Play Forward
Set Play Backward
Fast Forward/Pause
Fast Backward/Pause
Adjust Fast Forward Speed
Adjust Fast Backward Speed
Volume Up/Down

V. Annotation

a. Creating an Annotation

Pause the video at the time you wish to rate an activity or state. Right-click on the video screen to bring up the annotation menu. (Note: Right-clicking while the video is playing will automatically pause the video.)
  • Partial annotations are those under your "Partial Annotations" list on the right side of the Visualizer.
  • Recent annotations are those you have used selected recently, excluding those on the "Partial Annotations" list.
  • New annotation is where you can create a new annotation.

To make a new annotation, navigate the middle or last section of the menu until you find the activity or state you wish to add a certainty rating to. (Activities or states that are currently listed under "Partial Annotations" are grayed out.) Highlight it and select the desired value.


The new annotation should appear under the "Partial Annotations" list on the right.

b. Adding a Certainty Rating to an Annotation

Pause the video at the time you wish to rate an annotation. Right-click on the video screen to bring up the annotation menu.

Choose an annotation from the first section (Partial annotations). Highlight it and select the desired value. (Note: Adding "Not happening" will move a partial annotation to the "Current Annotations" list.)


You may also add a rating by right-clicking on the annotation, choosing "Add Rating" and selecting the desired value.


To add a rating to a current annotation, right-click on the annotation, choose "Add Rating" and select the desired value.


c. Changing or Removing a Certainty Rating

To change a certainty rating, right-click on the annotation that contains the rating you want to change and choose "Change Rating". Select the rating and the replacement value.


To remove a certainty rating, right-click on the annotation that contains the rating you want to remove and choose "Remove Rating". Select the rating you want to remove. (Note: If you remove all "Not happening" ratings, the annotation will be moved to the "Partial Annotations" list.)


d. Changing or Deleting an Annotation

To change an annotation’s label, right-click on the annotation that you want to change and choose "Change activity label". Select the new label.


To delete an annotation, right-click on the annotation that you want to delete and choose "Delete annotation".


e. Saving Annotations

The annotation data files will be saved when you make a change to an annotation or close the Visualizer.

VI. VideoMap Data Visualization

The VideoMap data view provides a time-locked visualization of sensor activations as they occur. As the playhead is moved, markings on the VideoMap change state to indicate where in the environment a sensor activation has occurred. Sensors are illustrated as simple shapes overlaid onto a floor plan on the environment where the data was recorded.


An Ekahau Position sensor is represented with a fuzzy semi-transparent colored circle where the system estimates the tag to be located at the present time.

The Object Movement MITes are represented on the map as small blue circles. Each time an ObjectMITes records an activation, the circle briefly flashes yellow. Passive infrared Motion MITes are shown as large gray semi-transparent circles that flash yellow when they detect motion within the approximate range represented on the map.

Double-clicking on one of the sensor icons will cause the visualizer playhead to move to the next activation time for that sensor, assuming it occurs within the current hour.


Right-clicking on the sensor icon will generate a pop-up timeline for the current day, with red tick marks whenever the sensor has been activated. Clicking on a region in the timeline will cause the visualizer to open the hour of data represented by that region.

VII. Time Series Data Visualization

In the area below the video display port, a zoomable time series visualization is used to present scaled values over time.

VIII. Bug Reports

Please report bugs to

IX. Credits

The BoxLab Visualizer was created by the House_n consortium at MIT.

X. Acknowledgments

Work on BoxLab and generation of shared sensor datasets using BoxLab is supported by National Science Foundation grant #0708375 (CRI: CRD Development of Longitudinal Home Activity Datasets as a Shared Resource). Equipment used in this work was donated by Intel Corporation, Microsoft Research, and Mitsubishi Electric Research Laboratory (MERL).