Making mixed reality: a conversation with Alexandros Sigaras and Sophia Roshal

Meet the team using mixed reality for cancer research

Welcome! This is Making mixed reality, a series celebrating the passionate community creating apps and experiences with Windows Mixed Reality. Here, developers, designers, artists (and more!) share how and why they got started, as well as their latest tips. We hope this series inspires you to join the community and get building!

Meeting Alexandros Sigaras and Sophia Roshal was a lot like mixed reality: a digital-physical fusion. It first happened through a flurry of tweets and emails as Alexandros, a senior research associate at Weill Cornell Medicine (WCM), and Sophia, a WCM software engineer, rapidly prototyped a Microsoft HoloLens application to achieve the Englander Institute for Precision Medicine’s “cancer moonshot,” a promise to empower better and faster cancer research, data collaboration, and accessible care. Soon after I was lucky enough to demo their project in-person. It’s now in the Windows Store as Holo Graph, an app enabling researchers to bring their own network data into the real world to explore, manipulate, and collaborate with other researchers in real-time, be they in the same room or on the other side of the planet.

Find out what makes this team tick, and how they make big data approachable with Windows Mixed Reality.

Why HoloLens, and why Windows Mixed Reality?

Sophia: It’s the logical next step. You usually constantly switch from window to window [on a PC]. With HoloLens, you stay in one place. You can just point at something; you don’t have to use your mouse. It’s just so much more of a natural environment, which is great.

The best part of mixed reality for me is seeing other people try it for the first time. They are surprised how well interactions between the real world and holograms work, and are excited to see new updates. The most exciting part is to see the endless possibilities of mixed reality. From games to medical research, there are still many applications of mixed reality to explore.

Alexandros: One of the key questions we get every single time we show HoloLens to someone who is already an avid developer is, “Why HoloLens, and not a 2D screen? Why does this revolutionize our work?” The key answer behind this is simplicity, connecting these dots. The amount of high-quality data that you can parse through with holograms is significantly more than the amount of data that you could create in a table and fuse together in your brain! Tangibility and collaboration are the biggest improvements. It’s like saying the mouse and the keyboard are absolutely great, but phone touch screens are a better user interface. We treat HoloLens as a technology that allows us to go to a higher level, make things more tangible, and remove the challenges of making connections in your brain because you actually see and manipulate them.

https://twitter.com/AlexSigaras/status/897242543276138496

Who uses Holo Graph?

Alexandros: In a nutshell, the end users for Holo Graph are computational biologists, clinicians, and oncologists. Instead of looking at “big data” in a two-dimensional structure, they immerse themselves and explore and focus on their areas of interest in 3D. There are two scenarios that we currently use with Holo Graph. One is for cancer research and genomics, and the other is metabolomics.

For cancer research, it’s for drug discovery. We want to find how specific drugs relate to specific genes. With our app, I can upload my network that has all of this correlating information, and I can explore it, manipulating and changing the ways I look at data. If I click on a hologram of a drug, I’ll see the drug’s most up-to-date information directly from the Food and Drug Administration (FDA) – that’s an API tool. If I click on the gene, gene cards will tell me more about that specific gene.

The other use case that we’re doing with our colleague Dr. Karsten Suhre from Weill Cornell Medicine in Qatar is metabolomics. Dr. Karsten Suhre has identified connections between metabolites, genes, and diseases such as Crohn’s disease and diabetes. Using Holo Graph he can browse and identify unexpected paths in the network. One of the latest videos that we shared was collaborative sharing and manipulation of this very network on Crohn’s disease to identify if there are unexpected connections to other diseases.

When did you get started building and designing for Windows Mixed Reality? Any tips for others just beginning? 

Alexandros: We became interested on the platform about two years ago and were delighted to be included in the first wave of HoloLens devices that shipped. Whether on an online forum or a meetup, there are a lot of talented people happy to help you get there and share their experience. Don’t try to reinvent the wheel. You will be surprised how many questions have already been answered before! As far as tips go, download and try out apps from the Windows Store and make sure to reach out to the community for any questions.

Sophia: Fragments was the biggest inspiration for me because the virtual characters sit on actual chairs. We recently updated our app to include avatars and do the same thing, and it was really cool to see that! Our avatars follow the person’s movement, and we also use spatial mapping to find the floor because the only point of reference on HoloLens is the head. There are MixedRealityToolkit scripts that finds planes, where you can find the lowest one and that will be the floor. Then you calculate the height between the head and the floor and you can map the avatar from there.

Alexandros: Doing tutorials Holograms 240 with the avatars and Mixed Reality 250 with sharing across devices are excellent examples to see the capabilities here.

Sophia: For someone just beginning, Mixed Reality Academy is by far the easiest way to start building apps. MixedRealityToolkit is the main tool I use. I will write my own scripts most of the time, but if you’re just starting, you have to get it!

What inspires you?

Sophia: Impact on our patients’ lives. Seeing your code making a positive impact in someone else’s life battling with cancer is one of the most rewarding experiences ever. The Englander Institute for Precision Medicine is using cutting-edge technology to go through a sea of data and provide our care team and their patients better treatment options. We believe that AI and devices such as HoloLens are just beginning to show their true potential, and we look forward on what’s yet to come in the near future.

Alexandros: And it’s all about the power of people. The headset doesn’t save someone’s life; the clinician does. But mixed reality helps them see the patterns and get there. With HoloLens we want to answer, “If I were to show you this before you made the call, would that change anything? Would it make your decision and response faster? Would it give you more data?” And every person that we ask nods their head and says, “Yes, it’s right there. It’s almost like I can touch it.” Clinicians are used to reviewing genomic reports that can span up to hundreds of pages requiring significant time and effort. This doesn’t have to be the case though. HoloLens can act as a catalyst by significantly reducing the review time and make an impact at scale when combined with other tools such as AI, machine learning, and deep learning that we also do at the Institute.

How are you getting data into your application? Any tips for those who want to do data visualization with HoloLens?

Sophia: The easiest way to load dynamic data into an application is through a cloud integration app such as OneDrive or Dropbox. When you share data across the network to other users you need to consider secure transfer and adopt standard formats. Holo Graph currently supports .csv and XML/GraphML formats on OneDrive. We tend to share data across using JSON.

OneDrive loading has been a great surprise. We used to add files to the backend. With OneDrive, now anyone who wants to can load their data into the experience.

Alexandros: Data and data privacy are of utmost importance to the Institute. The real value of Holo Graph is not just about its looks; it’s about empowering researchers to get their real data in securely. As far as visualizing the data, my tip is to enable your users to break out of the 2D window and put their data on their environment on their terms.

I’m ending with a favorite quote from our conversation. Spoiler alert: mixed reality’s got game!

Alexandros: The way I explain mixed reality is this: Imagine you have a virtual basketball. If you’re throwing it onto the real ground, it bounces off because it knows where the ground is, and there’s “friction.” You can repeat this 100 times and it would happen the same way – you can expect it. It’s literally bringing digital content into real life, allowing you to bend the rules.

Sophia and Alexandros are seriously inspiring. You can connect with Sophia and Alex on Twitter @RoshalSophia and @AlexSigaras.