We are developing a Virtual Reality work environment, and need help tying the Windows Desktop Duplication API into Unreal Engine 4.
The goal is to have an Unreal Engine project that mirrors desktop applications individually. Where VNC or Remote Desktop shows the entire desktop of the Host PC, I want to show individual applications on a 3D plane object. They should update and handle mouse and keyboard input just like VNC normally does.
Launch the UE4 application
Launch MSWord on the windows desktop as you normally would.
A 3D plane should appear in the UE4 world, showing the MSWord screen. It should update in real time, just as it would in VNC.
The user can click on the pane, and keyboard and mouse input will be captured and sent to MSWord, with all visual updates from MSWord being showed in real time on the pane in U4.
Launch another application on the windows desktop, say Notepad.
A new pane appears in UE4 showing the Notepad window.
The user can click on the NotePad pane in UE4 and keyboard and mouse input will be relayed to NotePad.
I have some working C++ code that does desktop duplication. Currently it just has a sample executable that demos the capabilities, I'm happy for you to use this code as a base or write your own. Ultimately I'd like the C++ code to be embedded fully within UE4, but the most important thing is to get the proof of concept working in UE4, however that can be acheieved.
I have basic C++ understanding and decent UE4 experience. I'm open to suggestions on implementation, just need to get this working in UE4.
Existing example C++ code for desktop duplication is here: [login to view URL]
Deliverables: Ideally I'd love to get an Unreal Engine 4.11 project that has it all working. But I'd settle for getting the C++ desktop duplication stuff working for applications and some help tying it together in UE4.
For reference, something like this is where I'm heading: This is kind of where I'm heading: http://blog.leapmotion.com/hood-leap-motion-hackathons-augmented-reality-workspace/
For now simple mouse interaction is fine. It's just getting the windows into UE4 and handling the input that I need done for now.
If you're interested in the job, please ask any questions you need and when satisfied, provide me with a brief outline of how you'd go about it, what the breakdown of tasks is, and your rough time estimations on each bit. This is important so I know we're starting off on the same page.