Video Manipulation Interface

I was heavily impressed by this demonstration of a Minority Report inspired image manipulation interface by Garratt Gallagher (for those wo haven't seen the movie here's the scene that shows the interface).


Today I'm trying to take it to the next level. Here's my version of such an interface that focuses on videos. It lets you place (predefined) videos in 3D space on top of the RGB image captured by the Kinect. A simple gesture lets you time-shift the video and another closes the video.

Written in c# using Kinect, OpenNI and WPF4 (this is the very first version and there's a lot to improve...)


These are the input gestures that are currently working

• Defining a video space with 4 fingers (2 on each hand) starts a new video inside that space
• Opening the right hand while pointing at a video with one finger of the left hand closes the video
• Opening the left hand puts the video into time-shift mode. Moving the right finger on the x-axis forwards or rewinds the video

The source code for the video manipulation is not online yet. The hand and finger detection can be downloaded from CodePlex: http://candescentnui.codeplex.com

Improving Finger Detection

The next version of Candescent NUI will find the direction the fingers are pointing at (2D for the moment). I just started working on this but the first tests look promising:

 

I also want to make the code more modular, so it will be possible to "plug in" different algorithms for the various steps without the need to work directly with the source code.

 

 

New Kinect Hardware

Microsoft has announced that there will be new Kinect hardware (including new firmware) next year when they launch the commercial platform. It's not clear whether the firmware of the current devices can be upgraded.

The most important feature for me will be the "Near Mode" that is optimized down to 50cm distance and should even work (though less well) at 40cm. Quoting Craig Eisler:

"This is one of the most requested features from the many developers and companies participating in our Kinect for Windows pilot program and folks commenting on our forums, and we’re pleased to deliver this, and more, at launch."

Thanks to those who supported my thread in the kinect forums!

Comment Comments

What do you do if the coding guidelines that apply to the project you're working on require that

- "around 25% - 50% of code should be comments"

- "every source file must have a history section at the top"

- "every method must have java doc or summary tags"

- "the ending bracket of a method must be commented with the method name"

Do you do it because you are required to? Or do you ignore that guideline knowing that you might get into trouble?

Luckily we were able to convince the right people that this makes no sense and that it's the guideline that needs to change. But unfortunately a bit too late: Today I still come across gems like the one below:

/// <summary>

/// calc rectangle with offsets

/// </summary>

/// <param name="index"></param>

/// <param name="offsetX"></param>

/// <param name="offsetY"></param>

/// <param name="left"></param>

/// <param name="top"></param>

/// <param name="width"></param>

/// <param name="height"></param>

/// <returns></returns>

private Rectangle CalculateRectangle(int index, int offsetX, int offsetY, int left, int top, int width, int height)

{

  [...]

} //CalculateRectangle

Enterprise Infrastructure Misconfiguration

I'm currently working as an external employee in a project for a government agency that also provides the hardware I have to work with.

This Monday I stopped the time it takes from the moment I press the start button on my laptop (running Win 7) until I'm logged in and Outlook and Visual Studio are ready and I can start doing some real actual work.

The result were shocking 16 minutes (down to 10 minutes today, but still...)

Let's do some math: 220 workdays a year * 16 minutes = 58 hours wait time, or almost 1.4 workweeks of lost time.

Why...?

Candescent NUI 10943

I've uploaded the new version of Candescent NUI to CodePlex (Binary and Source)

Changes

• Upgrade OpenNI from 1.3.2.3 to 1.3.3.6
• Upgrade Kinect SDK from 1.0.0.11 to Beta 2
• A lot of internal refactoring (but also including breaking changes on the interface)
• Some namespaces have changed
• Added (more) tests
• Source code for the Test Data Collector WPF application
• StartMenu must be configured to use either OpenNI or Kinect SDK (see app.config)

If something stopped working for you, please write to info@candescent.ch  I'll try to fix it over the weekend.

#region is evil

Today at work I came across this piece of code. Let's focus on the #region part.

public void Validate(Guid instanceId, Guid ruleSetId)
{
    [code here]
#region logging
        if (logger.IsInfoEnabled)
            logger.InfoFormat("Excecuting validation for instance '{0}' with ruleset '{1}'", instanceId, ruleSetId);


        if (logger.IsDebugEnabled)
            logger.DebugFormat("Validation script for instance '{0}'':\r\n{1}", instanceId, script);
#endregion
    [more code here]
}

When collapsed it looks like this:
    [code here]
+ "logging"
    [more code here]

We could improve it a bit by using a better region name:
    [code here]
+ "Log Validation Execution"
    [more code here]

But still: When the region is collapsed we don't see what's happening. It might be anything, because the code has access to all method and instance scoped variables. If the region is expanded we get lost in details.

What I would do:

1. Move all the logging code to a seperate class (say ValidationLogger)
2. Replace the #region with a single method call:

    [code here]
    log.LogValidationExecution(instanceId, ruleSetId);
    [more code here]

The advantage is that you see what happens (a method call) and what information will be used to log (the passed parameters) and it's also just one line.

Please, think twice before using the #region tag. In many cases it's a strong indicator that you need to refactor your code!

Testing the Code

I've run some code coverage analysis and here's the result. The basic test that loads a saved frame and then runs the hand detection has the following coverage: 

Code Coverage in Version 9489

CCT.NUI.Core                      38%
CCT.NUI.HandTracking         68%

The next version will have at least the following coverage

CCT.NUI.Core                      71%
CCT.NUI.HandTracking         72%

I reorganized some namespaces and also tampered with the interface, so the new version will require users to update their code.

I've also migrated to the newest Kinect SDK Beta 2. But because it still does not return depth values < 800mm it's not really an alternative to OpenNI.

I hope to be able to release the new version over the next weekend!

Edit (09 Nov 2011):
Let's see how high I can get it. Current values:
CCT.NUI.Core                      80%
CCT.NUI.HandTracking         72%

I won't reach 100% coverage in the core library because I can't test OpenNI and Kinect at the same time. Maybe I'll put the interface code in seperate assemblies (like CCT.NUI.OpenNI and CCT.NUI.SDK). This would also have the advantage that only the required dependencies are referenced and the other can be ignored.

New Test Data File Format

In the first version of the Test Data Collector I've used .NET binary serialization. This has turned out to be too inflexible.

v2 of the TestDataCollector saves the data as XML. The depth frame is encoded as base 64 byte array inside the XML. This results in a larger file size, but the hand data is human readable and no type information is saved within the file.

v2 can still open the old files. They can then be saved in the new format.

I've updated the download on CodePlex.

Warning: v3 will not be able to read the old binary tfrm files anymore, you have to use v2 to convert them!

Kinect SDK Beta 2

Microsoft has released the new version of the KinectSDK Beta 2:

http://www.microsoft.com/download/en/details.aspx?id=27876

The changelog says nothing about the minimum depth threshold, is it still 850mm?

Edit:
Eddy Escardo-Raffo wrote in the KinectSDK forum on October 24, 2011 that they have prioritized the request to make the minimum depth value configurable. The forum post ist here.

So maybe it'll be possible in the next version.

Edit 2:
I've tested the Beta 2 on my Laptop and it did not return any depth value below 800mm.


PS. Here's a interview with Rob Relyea about the new Beta. The interviewer might actually have mentioned Candescent NUI on CodePlex at the very end, though he didn't say a name :-)

Test Data Collector

I've published a first version of the TestDataCollector today. The program can be downloaded here (binary only for the moment):
http://candescentnui.codeplex.com/releases/view/76254

It currently only works with OpenNI.

When I have collected a first batch of data I'll define an accuracy metric that compares the human set points to those that the algoritm finds.



How to collect test data:

1. Click "Start Depth Source", this will start the depth data stream and show it on the screen.
2. Click "Capture Frame" to capture a single frame out of the depth stream (or click "Capture Frame Delayed", then the frame is captured after 3 seconds).
3. Click one of the frames you captured in the frames bar
4. Add as many hands as are presented to the device
5. Mark the center of the palm by clicking the button under the corresponding hand box and then on the image at the right location.

6. Mark the fingers by clicking the button and then the fingers on the image. Try to mark the point on the edge as shown in the image (but make sure it's still on the hand, look at the z position). 

 7. Save the frame by clicking "Save Test Frame..."

If you would like to contribute to the test data collection, please send me your captured frames in a zip file to info@candescent.ch

I have to add the following:

Please note! By sending me data generated with the test data collector you agree to transfer your copyright of that data to me. This allows me to publish it in a test data collection under any license. This is necessary so I can add it to CodePlex and / or to create a commercial version somewhen in the future (no plans yet).

The data contains no other information than the depth frame and the annotated hand data (no color image). No data about you or your computer is collected.

If you don't agree, please do not use the test data collector.