Matterport scan to game Engine2986
|
UserName private msg quote post Address this user | |
| Give me the ability to stand inside a Matterport model and command the walls to change color and you've got my money. Looks like you did it in your demo. Seems like that ability would open up a world of home improvement/home remodeling uses. | ||
| Post 26 • IP flag post | ||
Putten NLD |
DannyBasting private msg quote post Address this user | |
| Thank you all for those kind reactions. I got a little "mini-update" for those of you following this tread. As I mentioned before I've created a mini-map in an older project. I just copy-pasted the code from that and replaced the image. There is a little map in the bottom corner with a white Field of view indicator. As soon as you walk around the indicator moves to the corresponding location on the map. Of course it would be very easy to change the floor plan image. I just grabbed a quick screenshot from the Matterport space in this case. Sort video of it down below: Click here.  |
||
| Post 27 • IP flag post | ||
|
Jacques private msg quote post Address this user | |
Awesome this little map. Definitely!  I would love being able to do it as well. |
||
| Post 28 • IP flag post | ||
|
JC3DCX private msg quote post Address this user | |
| wow, this is awesome, wish I had the knowledge to do the same | ||
| Post 29 • IP flag post | ||
|
|
malfie private msg quote post Address this user | |
One can only imagine what @Metroplex360 and @DannyBasting could accomplish when they are put together in a room for a week!  |
||
| Post 30 • IP flag post | ||
Frisco, Texas |
Metroplex360 private msg quote post Address this user | |
| @malfie We're sort of working at the opposite end of the spectrum. @DannyBasting is being creative; I'm usually just tugging at threads that I discover and then making the results look pretty |
||
| Post 31 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
Anyways, if you're all done locking me and Metro up for an entire week, here is a little update  I mentioned using Riftcat's Vridge application to use my Samsung S6 device as a VR headset with Unreal Engine. Thought I'd show you guys what it looks like with a little recording. I don't have any decent recording equipment so please don't mind the amateur approach on this recording :P video here I can't wait to get my hands on a HTC Vive and experiment with roomscaling (walking around in VR) and motion controllers (to grab stuff)! |
||
| Post 32 • IP flag post | ||
|
|
JC3DCX private msg quote post Address this user | |
| Looking good, :-) | ||
| Post 33 • IP flag post | ||
|
|
UserName private msg quote post Address this user | |
| Good progress. Imagine a realtor with a "Vive" room. You go in there, put on your headset and walk around homes. Since you work with point clouds, you may find OcuTech interesting.  That's a screenshot I took while exploring an OUCtech point cloud model running in the browser. You can use your arrow keys and mouse to navigate that model as if you were playing a video game. Here's their news about OcuTech .. ------ "OcuTech360, a provider of 360° visual technology services, has announced a solution that enables users to extract 3D models and point clouds from video captured with spherical 360° cameras. Users can walk through an environment with a 360° video camera—or fly an environment with a video camera attached to a drone—and generate a range of deliverables. With a single pass, users can generate 3D models and point clouds, which can then be exported for use in 3D CAD software for applications including as-built documentation, clash-detection, retrofitting, remodeling, and more general documentation purposes." ---------- If that's all true, you could walk through a room using an ordinary 360 camera and OcuTech could turn the point cloud into a model you might be able to use. More importantly for me, they say you could fly a drone 360 camera through an environment to capture your video. If that worked out, perhaps you could film a room without walking -- you simply fly your drone around assuming you find a suitable indoor drone that carries a decent 360 video camera. Their online demo seems to take a while to load. It has a panel that lets you tweak the model to make it clearer. If an OcuTech point cloud is the same as a Matterport point cloud, that implies that you wouldn't need a Matterport camera to create a point cloud by walking through a house. Given one of those point clouds, maybe you play with it in your game engine the way you fed a Matterport point cloud into the game engine. Perhaps there's a catch to it since OcuTech doesn't seem to be competing in the home realtor market. They're targeting large organizations. You can read about OcuTech and other current 3D scanning news at this Spar3D site. With current stories about RealSense, SphereVision, Microsoft's VR plans and other point cloud technology, that looks like a good website to bookmark to keep up with what's going on in the 3D visualization world. Side-benefit: OucTech says that not only do they provide you with a 3D point cloud, but you get 360 videos of your walkthrough as well since you used a 360 video camera to film your environment. |
||
| Post 34 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| Looks interesting.. I'll have a closer look at it tonight! | ||
| Post 35 • IP flag post | ||
|
mori private msg quote post Address this user | |
| We are also working on this for Vive and the bigger version of Fraunhofer called HolodeckVR. I am also pretty sure that this is something which Matterport wants to achieve, especially for VR solutions. The scan and pointcloud quality captured with the current MP camera should also be much better than what you are allowed to download. Maybe MP can confirm this? Another issue with VR is that a detailed pointcloud model would go to the limits of smartphone headsets solutions like Samsung Gear. For great quality HTC Vive and a fast PC is needed. |
||
| Post 36 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| @mori, the XYZ data is pretty accurate if you ask me. Not to be confused with the OBJ that you can download from the workshop of course.. As for my UE4 update.. I could really use some feedback on the next addition. Currently I am working on a furniture menu. When I press the right mouse button, a menu appears. At the top of the menu you'll see a few categories such as "chairs / sofa's / tables". Clicking a category will display the available models for the corresponding category. After clicking a model, it will spawn in front of the user and the menu will disappear. They user can then walk towards the spawned model and rotate it. I was thinking of rendering some sort of indicators on which button to press to rotate clockwise or counter-clockwise as soon as the user is in close proximity of the furniture and can interact with it. I would also like to add the functionality of changing the materials on the furniture. Not a 100% sure on how to make this as user friendly as possible though. Anybody has any thoughts on how to create a more user friendly experience? Oh yeah.. the video below shows my first menu test. For some reason I couldn't capture my screen so I recorded it on my phone instead Video here |
||
| Post 37 • IP flag post | ||
|
|
UserName private msg quote post Address this user | |
| Great job .. I'd take it as it is. I'm not sure what you mean by click without knowing which controller you think the user might be using. If it was a game pad, I'd like to be able to rotate an object by moving my thumb left or right around the thumbpad. The "Land's End" method would be amazing if you're going the gaming route. In the Land's End Gear VR game, you gaze at an object and MOVE it by turning your head. Here's an excerpt from a review .. "Land's End also utilizes a pretty fun gimmick to simulate moving objects with your mind." Using the same concept, maybe you could make a selected chair rotate left my turning your head left or ROTATE it right by turning your head right. If you wanted to go a step further, you could probably MOVE objects as well via your gaze. You don't need any controller in Land's End since everything happens by gaze. "Land's End"  If you want to go way outside the box, imagine a user saying "OK Matterport." Matterport responds with "Yes?" and the user says "Change Material" which opens a panel showing material types as thumbnails. Each thumbnail has a number so you could say "3" to apply the third material to the selected chair. That's the drill-down approach. Speech recognition works quite well in Unity. In the Street View app, for instance, I can say a city, state and address and the app teleports me there. I say "Pikes Peak" and I'm teleported to Pikes Peak. If you're using "OK Matterport," we might as well have the ability to say "Den" to teleport to the den. Suddenly it becomes a real game, especially if you put some Pokemons in it. If you get all this working, I'll buy the game/VR tour. |
||
| Post 38 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| @UserName, thank you for your insights! As for the controller input.. For a desktop version I will mainly rely on mouse and keyboard input. But as it is very easy to add game-pad support, I will be adding that later on as well. Great idea about the speech recognition! Never worked with it before so I have to do some research on the topic. I'll definitely look in to that this weekend! |
||
| Post 39 • IP flag post | ||
|
Viewing private msg quote post Address this user | |
| @DannyBasting This is great work and exciting to see what people are achieving. In five years' time I can see this option being applied to every listing as standard. Just this year alone the progress has been that insane. I can picture a couple sitting at home on the sofa, both wearing a headset, exploring the same property simultaneously, deciding how they'd like to decorate it if they bought it. Susan: "Let's just see what that table would look like if we placed it in the dining room and made it glass instead of wood." Peter: "Dining room? I already put a pool table in there. It's staying there. Go and look for yourself." |
||
| Post 40 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| Hello again everybody! I've been rather busy lately so I haven't been able to work on my UE4 template file. In the mean while, I've received a lot of questions about how exactly I created the clean geometry from the scan data though. Reading back my original post with these questions in mind, I can see that some topics I covered could be a bit confusing and could do with some additional explanation. Through this post I'll try to answer the questions I've been asked regarding the conversions. First thing I would like to explain is the difference between the OBJ file you can download from the workshop, and the pointcloud which you can request free of charge from Matterport using this form (Free of charge while still in beta). A 3D model contains points located somewhere in a 3D space. A pointcloud is a file containing A LOT of these points. If you where to open a pointcloud in a texteditor you can see it the coordinates of each point along with the RGB value of the point. Here are a just few lines from the pointcloud I used in my example: -5.073570 2.938401 2.786698 141 135 124 -5.073570 2.941165 2.791003 129 124 115 -5.075639 2.938401 2.791003 134 128 117 The first value in each line being the X-axis, the second Y, third Z. The other values being R G B respectively. The actual pointcloud contains almost 5 million lines (so the exact same amount of points in 3d space). If you where to connect those points (referred to as vertices) with lines (edges), you can create 3d planes (faces). See image below.  Here is an example of the OBJ file:  And here is an example of the pointcloud:  So what does that actually mean for us? The OBJ's that are downloadable from the workspace are great for the purpose they are created for (the dollhouse view in Matterport tours), but they are close to useless to do some professional 3D-modelling with. Besides having a lot of unnecessary faces on flat surfaces and being a bit messy over all.. the UV maps are close to impossible to work with. UV mapping is the 3D modeling process of projecting a 2D image to a 3D model's surface for texture mapping. Again, I am not hating on the OBJ files from Matterport, but editing them is just not what they are intended for. The good news is that Matterport (like I mentioned before), can provide us with pointcloud data from our scans. And these files are amazing. While you can't just press a button to magically convert a pointcloud in to a clean CAD model, the pointcloud data will help create CAD models faster and more accurate! I hope this long post will clarify some of the questions I've been receiving. I'll try to respond to all the questions I got, but I do apologize if I forgot someone, or if it takes me a long time to respond. Also, I'll try to get working on the UE4 template soon and post an update over here. Have a nice weekend guys! Dannny |
||
| Post 41 • IP flag post | ||
 WGAN ForumFounder & WGAN-TV Podcast Host Atlanta, Georgia |
DanSmigrod private msg quote post Address this user | |
| @DannyBasting You are awesome. Thank you for this great explanation about the data that Matterport captures and how it may be used. The MAGIC of Matterport really is the data that that the Camera captures. Once Matterport makes its APIs/SDKs publically available we will see amazing tools created by third party developers like you. The future - and the Matterport APIs/SDKs - can not arrive soon enough. Dan |
||
| Post 42 • IP flag post | ||
|
|
SVM private msg quote post Address this user | |
| @DannyBasting, thought you might be interested in this as it's very similar to what you're working on. I've been using their beta for a bit, they appear to have just gone live with their service. https://www.shapespark.com/#demo |
||
| Post 43 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| @SVM Thank you for sharing. Interesting application. While I do miss the ability to interact with the model it looks pretty amazing for a web based application! |
||
| Post 44 • IP flag post | ||
|
|
neeravbm private msg quote post Address this user | |
| Great work, @DannyBasting and thanks for sharing the process with us! I am working on a software to automate the conversion of point cloud to CAD model. I have a few questions: 1) How did you convert point cloud to CAD? How much time did it take? 2) I am assuming that once you created CAD models, you had to apply material to each one individually. Is that correct? 3) Did you also add lighting manually? 4) You mentioned that Pointcloud from scans are cleaner than the OBJ files. I don't have a matterport camera so may be my question is naive. What is the difference between point cloud and OBJ files? I thought OBJ stores point cloud. |
||
| Post 45 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| @neeravbm If you are able to automate that process and are able to create a model that I'd consider a good usable CAD model, you'll have a game changer at your disposal . I'd be first in line to sign up for that kind of service. That being said, I don't see anyone pulling that off any time soon for a couple of reasons. However I always like to keep an open mind and if their would be a possibility for anyone to automate that process, it sure would make my life a lot easier. So I'd be more then happy to discuss this further. Let me try to answer those questions for you first though. Quote: Originally Posted by neeravbm About an hour of 2 per floor, give or take. Quote: Originally Posted by neeravbm No, all tools in Archicad come with a lot of preset materials and composites. Once you pick one and start drafting all meshes will have materials. If you want to change a wall material for example its a simple 2 click process. The reason you don't see any materials in the CAD renderings is because I set it to clay render. Which renders everything grey. Quote: Originally Posted by neeravbm In the clay renderings? No.. just the standard sunlight without any tweaking. Quote: Originally Posted by neeravbm Please read my post above from about a month ago. The one with the green cube image in it. This explains the difference pretty well I think. OBJ does not store any pointcloud data. Hope this answers your questions.. I've got a couple of questions for you as well. 1) What file format(s) are you planning to support for exporting? 2) How accurate do you think it could be? 3) Will you be creating textures from the pointclouds RGB data? 4) What are your thoughts on the UV-mapping process? 5) What file format for the to be imported pointclouds do you want to support? |
||
| Post 46 • IP flag post | ||
|
|
neeravbm private msg quote post Address this user | |
| @DannyBasting Thanks for your quick reply. Here are the answers: 1) What file format(s) are you planning to support for exporting? I was planning to use SketchUp format but that is being discontinued so I'll have to use some other format. What is your preference? 2) How accurate do you think it could be? The accuracy of the floor plan and placement of walls will be dependent on the accuracy of the point cloud provided by Matterport, which means that it will be very accurate. In the first iteration, I am just planning to recognize the object and get a corresponding 3D model from 3D Warehouse. This means that objects themselves will not be very accurate. Is this a hindrance? 3) Will you be creating textures from the pointclouds RGB data? In the first iteration, there will be no texture mapping. It will look like clay render. In the next iteration, walls, flow and ceiling will have textures, obtained from the point cloud. Texturizing objects is more difficult because is user moves the object, it is possible that unscanned portion of it may become visible. In that case, you have to predict texture and that becomes difficult. The only way I see right now is use machine learning to predict texture, but this is complicated. 4) What are your thoughts on the UV-mapping process? For floor, ceiling and walls, it will be very easy since these are all planes. For objects, it most likely will be using machine learning, so I don't have a good answer for UV mapping for objects right now. 5) What file format for the to be imported pointclouds do you want to support? To start with, I'll support OBJ and PCD but am pretty flexible with this. The only reason I am starting with OBJ and PCD because I am familiar with these formats. If you think that any of my approaches above will cause a hindrance in your workflow, please let me know. If technically possible, I would like to make the application as easy to operate as possible. Questions for you: 1) Why do you think you don't see anyone automating the process of converting point cloud to CAD? I know that it's difficult but I would like to see your reasons to make sure that I am not missing anything. 2) My initial target market for this application was interior designees so that they can move furniture around and show to the client how the new layout will look. This is the first time I am hearing about its application in game design. Do you think that game design is a big enough market for this that a separate application can be created that caters to the game designers' workflow? |
||
| Post 47 • IP flag post | ||
|
|
neeravbm private msg quote post Address this user | |
| @DannyBasting For some reason, I thought you were using the CAD model to game design but that's not the case. So please ignore my second question above. From what I understand, you are a real estate agent/broker, you are trying to sell the house to your client and you want to provide an interactive 3D model of the house so that they can play around with it before they buy it. Is that the correct use case? If this is the case, how important is the visualization for you? In other words, won't converting point cloud to CAD model make it more game-like for the client or do you think client will still be interested in this? |
||
| Post 48 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| @neeravbm Thanks for your detailed reply. Quote: Originally Posted by neeravbm I've never been a fan of Sketchup myself. While it is super easy for anyone to create something in Sketchup, it can be tricky to create something usable with a bit more complex geometry. I have to work with Sketchup files from clients from time to time.. Since they lack the basic knowledge and skills of 3D modeling, I have to spend a lot of time cleaning up the models to something I useful for my visualizations / animations. But I digress.. I myself am a big fan of the FBX format since it supports just about anything from textures to animations etc. If you where somehow able to export it to a usable IFC format however, You'll have that game changer like a said in my other reply. Just to clarify, IFC is the worldwide standard fileformat for BIM applications like AutoCad, Revit, ArchiCad etc. You can read more about it over here. Quote: Originally Posted by neeravbm With objects, are you referring to furniture and such? I myself couldn't care less about those to be honest. As long as the building with all of its beams and columns are accurate, I'd be happy. Quote: Originally Posted by neeravbm I wouldn't use anything generated from the OBJ file that can be downloaded directly from the Matterport workshop. It is simply to low resolution and inaccurate for my needs. Like I mentioned in one of my original posts, the OBJ Matterport provides out of the box was never intended for that. It's created for for dollhouse view in the tours, for which it is great. Generating your own OBJ from the raw pointcloud data as a starting point might work, but If you can create something straight of of the pointcloud data itself, I'd go for that. Quote: Originally Posted by neeravbm Let me rephrase that.. I don't see anyone automating that kind of process to a point that it will be up to my standards. I do architectural visualizations / animations, as well as some BIM drafting. If I have to fix up a lot of work from the automated process it is often faster to just re do it. That's just not worth it at the moment. Quote: Originally Posted by neeravbm I am not a real estate agent or broker. I mentioned this somewhere above in this post. But I work for architectural companies and interior designers. I create 3d animations, stills, and interactive applications in order for them to convince their clients. Most of those clients are remodeling a house and want to see what it looks like beforehand. Here is an example of something I am currently working on, it's not finished yet though. Click here |
||
| Post 49 • IP flag post | ||
|
|
neeravbm private msg quote post Address this user | |
| @DannyBasting That's an awesome visualization! I am extremely impressed! From what I understood, it will be helpful for you to have a tool that converts the Matterport point cloud to a CAD model in IFC format. This CAD model will have floor, walls (including columns), ceiling, doors and windows, all without texture. You do not need CAD models of the furniture placed. Am I accurate? One more question: Do you even need windows and doors in this CAD model or will you be fine adding it yourself? The reason I am asking is that some point clouds seem to have closed doors and without image recognition, it will be impossible to figure out that it's a door and not part of the wall. So in first iteration, it will be easier to omit it. |
||
| Post 50 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| Thank you @neeravbm Quote: Originally Posted by neeravbm That would be accurate yes.. Quote: Originally Posted by neeravbm I do need those windows and doors of course, but I can easily add them in a IFC type wall. |
||
| Post 51 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| Hi all, It's been a while since I've posted an update over here due to a busy schedule. I've kind of abandoned the demo apartment project on which this thread started, however I've kept working on the UE4 template in general. I've made some additions to it and am close to "finishing" it. Just need to do some polishing at this moment. Those that have kept an eye on my Vimeo channel may have already seen the color picker I've added. You can see part of the template in action in my Archilogic project over here. |
||
| Post 52 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| Hi guys, I am close to finishing some projects that took up a lot of my time lately. So for those interested, I plan on doing a couple of updates over here shortly. I am also going to focus a lot on VR content. Yesterday I picked up this amazing device:  The idea is to create a virtual lobby as a main menu. Where users can; - watch a video file of the corresponding project(s) on a large TV screen - pick up a flyer of some sort, with project information - browse through project related images - enter a door and step in to the actual project Inside the project, users should be able to pick up items, (including furniture) with the use of motion controllers. Change wall colors (like I already implemented in my regular UE4 template), and much more. If any one has access to an HTC Vive or Oculus Rift and would like to test some of my upcoming VR content, feel free to contact me. I could always use some feedback. It's going to be a busy weekend with a lot of exciting VR testing. These are definitely exciting times to be working in the CG branch! I'll keep you guys posted on the progress |
||
| Post 53 • IP flag post | ||
|
|
neeravbm private msg quote post Address this user | |
| @DannyBasting I have an Oculus Rift (without the Touch controller) so I should be able to test. Let me know when you want me to do the same. | ||
| Post 54 • IP flag post | ||
|
Putten NLD |
DannyBasting private msg quote post Address this user | |
| Well I've been messing around with it for a couple of hours now and I absolutely LOVE it. This device is truly amazing, and far from just a gimmick. I am convinced VR will become mainstream (especially in my line of work) in the near future. The "problem", with VR at this point, is that video recordings don't do it justice. Neither do cheaper options like Google Cardboard and GearVR. While those do have their uses, you can't really compare them to the higher end devices which require a decent PC. Put a client inside a project using a device like the HTC Vive or Oculus Rift, and I am sure he/she will be sold on it. While videos doesn't do it justice, it is the only way to show it without putting people inside VR. So below a video of my test project. It's going to be an awesome weekend, testing out more ways to interact with my projects inside Unreal! @neeravbm I'd be happy to send you a test project but without touch you'll be limited in functionality, since everything is mapped to touch controllers. |
||
| Post 55 • IP flag post | ||
This topic is archived. Start new topic?