This is part two of a series (part one); most of the technical detail will be in the last post. Between NDAs and work disclosure, I can’t release any code, but I can discuss concepts, caveats and successes in the hopes that it helps someone else as much as the open source Kinect projects & the Kinect for Windows community as a whole has helped me.
I had gotten the green light, so now this was exploratory. I decided the best place to start would be with some of the toolkits and samples that the community had created. I grabbed KinectNUI & the Kinect Toolbox. KinectNUI has a lot of legwork done in gesture detection, smoothing & interop with traditional devices (mouse/keyboard). The Kinect Toolbox has a really sweet templated gesture detection engine, as well as a playback/record system that lets you debug without jumping out of your seat (ah, laziness IS the mother of invention).
Windows 7 + WPF + Kinect Public SDK
I started out extending the KinectNUI – my experience with WPF has been limited, so it was a good opportunity to sharpen that skillset. I wiped most of the KinectNUI out, leaving me with the old ‘MainWindow,’ which was now a ‘diagnostics’ window. In fact, it was now a user control, as well as all of my other content pages – so I could get some animations between controls to give a more ‘Xbox Dashboard’ effect.
MainWindow
It started simple enough – I made my XAML window the same size as the screens our prototype would be running on – 1080p HDTVs. The camera was pumped to a live-view canvas in the bottom corner (a la Dance Central), our logo was in the top left corner, and our main content viewport was a canvas in the middle with some animations for swapping user controls. The ‘MainWindow’ root canvas had the Kinect hands painted on top of that whenever a Skeleton frame was ready.
Tiles/Canvas
So our main content viewport is the good stuff – all the content goes here. Tiles with actions across the screen; hovering on a tile (or clicking) executes the action associated with that tile. Quite simple really – I wrote a base class - we’ll call it ‘ASweetKinectTile' – which inherits from UIElement. My tiles inherit from ASweetKinectTile, with some override-able methods (say, SwipeLeft, SwipeRight, Click, etc), making virtually any element Kinect-enabled.
Hover-to-Activate
This proved interesting – and simply came down to mapping the coordinates of the ‘tiles’ on the screen & counting skeleton frames while a specific joint (or combination of joints) was inside the bounds of that element. I’m sure some WPF wizard could make it happen in some ‘proper’ way, but this was rapid prototyping – working with temporary hax is a lot better than ‘not working, but no hax’ – anyway, it worked quite swimmingly for a while.
Gestures
While the gesture detection in the KinectNUI worked fairly well, I read some good things about the TemplatedGestureDetector in the Kinect Toolbox, so I decided to give it a shot. I will say this now and probably later. Gesture detection is a pain. Not the detecting gestures part, but the ignoring inadvertent movement part. I’ve been noodling a couple of ways to avoid this (particularly with left/right swipes – the tendency is to swipe over, then move your hand back to swipe again, essentially swiping back in the other direction) – namely, a delay between detections.
Some Thoughts
While the solution was good, we liked it, the business people liked it, the UI was still far from polished – it was a prototype, after all (and only the first sprint). So I got to thinking: Tiles = metro. Full screen = metro. How about porting over the prototype into Windows 8? It seemed a natural fit. Next post – Windows 8 & a Metro app.
This is part one of a series (part two); most of the technical detail will be in the last post. Between NDAs and work disclosure, I can’t release any code, but I can discuss concepts, caveats and successes in the hopes that it helps someone else as much as the open source Kinect projects & the Kinect for Windows community has helped me.
I started writing this and realized it’ll be long. So I’m going to break it up into sections. Section one, the backstory.
My current project at work is pretty sweet. I’m to build an interactive system, accessible and easily usable to guys in gloves – i.e., no touchscreens – to manipulate project data, project images and project plans. Sounds easy enough, huh?
Starting out: Touchscreen
When I initially heard about this project, there was a lot of chatter around using a touchscreen. Let’s review why that’s a bad idea:
- Environment: construction site. I think this says it all:
- ‘Oh, see, just look right here – oops.’ Drill through screen.
- Cost: Touchscreens the size we were looking at (42”+) are expensive. I mean expensive. Especially with a shortened service life due to environment. Like $3k. $3k + 6-8 month service life? No thanks.
- User experience: the majority of our users are going to be construction workers. What do construction workers do? They build things. They also wear gloves. Gloves + touchscreen = no worky.
Touchscreen Alternatives: Kinect
Kinect popped into my head, what with the dev community doing the cool things they’ve been doing – I expected to be looked at like I was crazy. After the initial questions, business people started throwing around the words ‘Minority Report’ – tip: that’s an exciting time. That’s when business need meets holy-f*n-hell-this-is-sweet dev work – it’s win-win career gold. Champagne falls from the heavens, velvet ropes part – the works.
Here’s what makes the Kinect a great tool for this application:
- Environment: construction site. I think this says it all:
- Enclosed plexiglass box. No unsolicited poking. No accidental impalement.
- Cost: a 42” 1080p TV can be scooped up cheap now. I think the one we found was around $600 for an LG LCD.
- Kinect Sensor: $150. $99 over the holidays (although with the Kinect for Windows announcement, that looks to be going up to $249, but $100 is cheap for a commercial license – so stop whining about the cost differential)
- $600 TV + $150 Kinect = $750 - AND the service life expectancy is greater. Fewer replacement cycles at lower cost.
- User experience: besides the fact that it’s sweet as hell, it’s natural and easy to pick up, from the project exec in the trailer to the guy welding massive steel beams together.
Bonus! Voice.
There’s something else we get for free with Kinect – voice. The Kinect microphone array is killer, so while we haven’t done any field testing, it’s expected to be able to compete at some level with a lapel mic.
I’m at the SharePoint Conference in sunny Anaheim CA this week. First day of sessions is in the books. Keynote and most sessions are heavy on cloud services (particularly Office 365) and end-user empowerment. Not seeing a whole lot of developer-specific sessions this time around. Attendance is similar to the last trip through, around 7500.
This morning, 20 minutes before the keynote, a company called huddle showed up with a marching band. And cheerleaders. And signs and flyers featuring ‘SharePoint’ with a line through it (a la no-smoking signs). I can’t speak to their product’s legitimacy as a true SharePoint competitor, but I can say that their marketing department is ballsy. I mean, showing up with that kind of fanfare at one of the biggest conferences of the year? That takes some fortitude. To that end – watching people in black shirts descend upon them (complete with SharePoint logos and talking into their collars) was definitely a new level of entertainment.
I attended three sessions today – the first, past the keynote, was focused on Knowledge Management and creating knowledge communities. The session was definitely business oriented, but shed some light on what other businesses are struggling with to generate user adoption of knowledge systems. There are a lot of sessions, blogs & books out there about increasing user content into SharePoint, but no one really focuses on innovative ways to get that data back out. That’s a void that I imagine the community will fill soon, but it’s a big opportunity, and one of SharePoint’s biggest ‘out-of-the-box’ weaknesses.
After lunch I went to a session that had ‘iPad’ ‘Android tablet’ and ‘Windows Devices’ in the title. I was expecting something sweet. I was disappointed. It was basically OCT (Office Customization Toolkit) 101 – and how to tell your users ‘NO.’ Had to bail early on that.
I couldn’t really find a third session that looked useful today. As I said earlier, there is a lot of focus on O365 & end user experience. Ultimately, I started out in document management, but bailed and had a few phone calls and conversations that were far more encouraging.
So day one wasn’t bad. Here’s hoping day two has some more dev-central things, but that’s ok if it doesn’t. End user adoption is important, and something we should all embrace, developer, administrator or business user.
I’m floating around – normal looking guy with a beard in jesus sandals. Say hello.
In case that rock you’ve been living under is particularly dense, the Windows Phone Mango beta 2 update has arrived for registered developers, along with the 2nd beta of the Windows Phone Developer Tools.
You can get the dev tools here: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=26648
And if you’re a registered developer, you should have gotten a Connect invitation sometime around mid-day. My update process went fine, and took about an hour.
I installed Small Business Server last weekend here at the house in an attempt to consolidate some boxes and clear some clutter in preparation for the impending move. A really sweet part of SBS is the Remote Web Workspace, which gives you links to all of the internal resources SBS publishes – you get links to OWA, SharePoint, Shared Folders & Remote Desktops. Plus, it’s all extensible, so you can write your own providers to publish whatever data and widgets you want up to your users.
Since I don’t really have ‘users’ per se, just me and my wife, I don’t really care about UI customization. It looks fresh and modern, like the Windows Live redesign. Functionally, I love it – it takes all available resources here at my house and packages them up in an easy to use interface.
One part you’ve got are the shared folders. The interface is spartan, but usable – you get a tree view & a search, and you can download whole folders with two-clicks – it will auto-zip them and fire them over the wire.
Great, you say – until you realize that it’s only good for shares on the local machine (i.e., the Small Business Server box). I have two servers – one running SBS, the other running Server 2008 R2, with Hyper-V and lots of cheap storage. Since almost all of my storage is on the other box, this presented a problem. The SBS Console refuses to add anything but locally-attached drives. I tried all kinds of stuff, from Linux-esque symbolic links to NTFS junctions, none of which worked.
The more I thought about it, though, the more I realized there had to be a way to get the shares from my other server exposed this way as well…I mean, why have the server as the top-level of the tree view if it’s always the only root?
There is no documentation on this, so I got out reflector. I could go on for an entire post about how RedGate is an awful company for charging now for reflector, but instead I’ll just say this – go get ILSpy.
Before you get into any of this, remember, if you are in a production environment, please test much more thoroughly than I have. I’m only using this with <5 users and it is certainly not a ‘production’ environment. If you’re doing real work, by all means, be careful. This is horribly unsupported and chances are high it would break if service packs or anything broke the underlying implementation. There’s a reason most of these classes are private and internal.
Method A: Keeping it Simple
Find the web.config – it is here: Program Files\Windows Small Business Server\Bin\WebApp\RemoteAccess
Further down, you’ll find the storage provider (among a host of other providers):
<wssg.storageProvider type="Microsoft.WindowsServerSolutions.Web.Storage.SBSStorageProvider,
Wssg.Web.StorageProvider" />
Which is the standard implementation in SBS 2011. Some investigating yielded that it inherited from IStorageInformationProvider.
Some more digging showed that there were some other implementations of IStorageInformationProvider, one of which used a configuration key called 'shares.’
That one would be the FileSystemBasedStorageInformationProvider, (in Web.Internal) which pulled a list of shares from semicolon delimited list in the config file, from a configuration key called ‘shares.’ It does this in the Initialization method – choosing to use the configuration key if it is not null. This sounds well and good, BUT it is an exclusive or. You’ll get config entries OR file shares…meaning the standard pull from WMI won’t work any longer. With this method, you’ll have to explicitly define each and every share on any machine you want to appear in RWA.
Switching to the FileSystemBasedStorageInformationProvider is simple enough – remove (or comment out) the original line, and add this one right beneath it:
<wssg.storageProvider type="Microsoft.WindowsServerSolutions.Web.Storage.FileSystemBasedStorageInformationProvider,
Wssg.Web.Internal" shares="\\servername\sharename;\\servername2\sharename;\\servername3\sharename" />
Editing the web.config will fire off a re-JIT, so you’ll have to re-sign in. If you did it correctly, you should now see your shares – but only the ones you explicitly added to the config file.
Method B: Combining the Best of Both – Building a new StorageProvider
Since I had reflector out and I was already digging through Internal classes anyway, I decided to try and combine the output from both the configuration class and the SBSStorageProvider.
Needless to say, collecting dependencies is always the worst part. First, install the Windows Server SDK (you can get that here). That will give you some of them. The rest you’ll have to grab out of the Program Files\Windows Small Business Server\Bin\WebApp\RemoteAccess\Bin folder. Here are the ones you’ll need:
- Wssg.Web (SDK)
- Wssg.Web.Internal (from the bin)
- Wssg.Web.StorageProvider (from the bin)
- StorageCommon (from the Small Business Server bin - Program Files\Windows Small Business Server\Bin)
Unfortunately, the constructors for these are internal. There is probably a very good reason for this, but whatever – this meant I had to basically copy\paste the disassembled code into my project to get constructors (gross). It is gross, and should have been yet another red flag that this was a bad idea. It was not.
Getting the config file reader to work was pretty much a cake walk. It is a simple implementation and didn't take much to get into my project. The next one, however, was a bit more of a pain. The basic architecture is this: IStorageInformationProvider returns a generic List<IShareInfo>, which contains all of the information for the shares. The IShareInfo interface is pretty simple too. The concrete class StorageAPIBasedShareInfo, however, is not. It has dependencies all the way into the StorageCommon assembly, which is why it is referenced. Anyway, what these things do is pretty simple – get some shares, check some permissions, and, ultimately, return a generic List<IShareInfo>, which then gets returned out to whatever requesting consumer.
I created my assembly, which you can download below. It basically calls the shamelessly copy\pasted\tweaked methods from the two StorageInformationProviders and combines them before sending them back out. All of the caching and permissions mechanisms are still in tow, but again – please test before using this in any kind of production environment. I doubt it’s terribly kosher with the SBS peeps either.
Method B-2: Installing My Provider
Download this assembly: http://johndandison.com/stuff/sbs/johndandison.SBS.StorageProvider.dll
Or get the C# source here: http://johndandison.com/stuff/sbs/SBSAndConfigBasedStorageInformationProvider.cs.txt
Start in this folder on your SBS machine: Program Files\Windows Small Business Server\Bin\WebApp\RemoteAccess\
Copy the assembly to the bin folder.
Drop this line into your web.config:
<wssg.storageProvider type="johndandison.SBSAndConfigBasedStorageInformationProvider, johndandison.WSS.StorageProvider"
shares="\\server\share;\\server\share" />
That should be about it. Your box will need to re-JIT, but it shouldn’t take but a few seconds.
Addendum: Search
Search doesn’t work. When I have some time to rip the search assembly apart, I’ll attempt to write a search provider that uses all the shares, not just the standard SBS ones. Right now, it won’t search beyond the shares on your SBS machine.