Data Processing Instructions

I. Sensors
II. Video
III. Audio


Sensors

1. MITes data comes in binary (.b) format that needs to be converted to a human-readable (.csv) file. To do this:
  • - open C:\Users\HouseN\Desktop\BoxLabSubversion\MergerLense100324\HandLense\HandLense.sln and select the green arrow to run the program
  • - in the program, go to "Open Data Folder" and select the folder a level above the day file (this should be the kiosk level file)
  • - to check that the program has run successfully, go inside day folders and look for two files ending in MotionMITes.csv and ObjectMITes.csv

2. To merge the MotionMITes.csv and ObjectMITes.csv files from different kiosks, go to C:\Users\HouseN\Desktop\PanSubversion\projects\MITesDataConverter and double-click the .sln file, selecting the little green arrow to run it. You can drag and drop the highest level folder, and the program will pair the appropriate files (it compares timestamps within the .csv documents). It will output a "merged" folder next to the kiosk-level folders with merged MotionMITes.csv and ObjectMITes.csv within.

3. To add headers to the merged data, run (specifying files by editing the script, currently do once each for Object and Motion MITes):
C:\Users\HouseN\Desktop\BoxLabSubversion\BoxLabDataUtilities\trunk\BoxLabColumnHeaderGenerator

4. To organize data appropriately, run ConsoleApplication2 (on desktop).

End result should look like this:
SensorFolder -> day folders -> MITes folder -> ObjectMITes.csv & MotionMITes.csv


Video

Starting data should look like this (but may be further along when you find it):
Containing Folder --> day folders --> hour folders --> VideoFolder (also AudioFolder) --> KioskView Folders (Video0, Video1) --> 10 minute (sixths) zip folders --> jpeg files

1) Change Directory Structure

-- AVRenamer: within the hour level folders there are video and audio folders, and within those there will be view folders/microphone folders, originally labelled Audio0, Audio1, Video0, Video1. This script gives them a kiosk view ID (ex: 231) so that those folders can coexist.
-- in upload folder, copy over full directory structure for one kiosk and then copy just the kioskview/microphone folder & subfolders of other two kiosks using this script [link to uploaded script]

2) Processing Files

-- Unzip the images using [link to uploaded script]

-- Image Pruning (BoxLabImageFPSPrunerTEST.py)
  • -eletes images which are too noisy while retaining at least one image every 10 seconds (can be altered via SAVE_INTERVAL variable)
  • outputs an ImageFPS.csv or ImageFPS.log which has two tab-delimited columns representing time (format:YYYY-MM-DD HH:mm:ss.fff) and frames per second
  • compresses images?

-- Anonymize? (C# - Data Utils)
-- Rezip?

End result should look like:
AudioVideoFolder -> day folders -> hours folders -> video folder (audio folder next to it) -> kiosk view folders(1) -> 10 minute (sixths) zip folders AND ImageFPS.csv(2) -> jpeg files

NOTES:
scripts generally in: C:\Users\HouseN\Desktop\BoxLabSubversion\BoxLabDataScripts\trunk


Audio

Original Version:
stuff --> AudioFolder --> Kiosk Level (0 or 1, not labelled) --> up to 6 (0-5) zip files and an AudoStats.txt -> ~600 .wav files per zipped folder

1) Unzip
Use BoxLabAVUnZipper.py in Scripts

2) Sew (Seamstress .sln)

3) Shuffle (ShufflerByFolder .sln)


Final Version:


- We take the original files and thread them together to make hour-long files. It shuffles around audio in segments of 250 ms or less so that the audio isn't intelligible but you can tell background sounds.

-- At end, copy original audio & video data into restricted folders.



This doesn't belong on this page, does it? Unless you want to create a meta-section.

Data Structure:
E:\BOXLAB_ORGANIZATION_FOLDER
1. Sessions (name: (ex) BLP_1984-12-10_2011-10-28) usually weeks-months, determined by data collection landmarks such as the introduction of a new type of sensor)
a. readme showing what the folder currently contains (data types, dates/times, kiosks)
b. unprocessed data tree (move data into here as found, very descriptive folder names)
c. processed data tree using this data structure (when moved in here, ready to be visualized & subsequently zipped)
d. zipped file for website

!! Upload scripts to git or other page and link from here, rather than giving full paths.