One on One: A Conversation about Augmented Reality & Mixed Reality
I n the year 2007, it was a year of new and eye-opening possibilities, from Apple’s revolutionary iPhone to Microsoft’s attempt to patent EGG usability testing. Among the emerging tech, I became fascinated with one particular creation that was not actually an invention but more of an experiment. Johnny Lee, a human-computer interaction researcher who had worked on Microsoft’s Kinect development team, made an impressive demo extending the functionality of the Wii remote controller. Not only there was no additional hardware needed, he showed a cheap and efficient way to display VR right in your living room just by using the sensor bar and wii remote. You didn’t even have to be a developer to the run the sample demo. You would usually see VR displays at Siggraph or at other high profiled conventions that always seemed out of reach for the typical consumer. This was the start of the first step towards accessible virtual reality for the average consumer. One of first steps I believe that led us to where we are today. This wasn’t the origin of VR but it was such a quick and efficient hack shown within an accessible device that quickly became popular in the tech world. Due to the rapid popular of youtube, this demonstration was available to a much wider reach of audience and also how someone like myself came across it.
Even though this was a pinnacle moment that had stuck with me throughout my career, It will still be quite while before I actually entered into the field of AR/VR. At the time there were no “jobs” that had existed since no one was incorporating that tech yet for consumers. This was also the age of smartphones, so it was expected for a designer to pursue in advertising and agency type of work. Though never really satisfied, I caught a break about 4 years and got placed in an augmented reality mobile firm called Blippar. Over the years, I had became a lead designer and worked with big name brands from around the world. I credit a lot to my major that had incorporated many fields into our curriculum such as animation, motion graphics, 3D, code, and graphic design. This had prepared me toward the right mind-set and spatial awareness needed to execute my projects efficiently. Having these assets also made my communication immensely easier with all team members involved. Throughout the four years, I had worked with big and small brands on delivering high end campaigns involving both AR and VR. Whether delivered on a pepsi bottle, from the billboards on Times Square, or at Hennessy event, I have been through hundreds of pitches and projects from the concept stage all the way to a shipped out product.
This tech advanced to a point where it has peaked the interest of many current mobile/2d designers and non-designers alike. As someone with past experience, I get asked a lot of the same questions about VR, AR, and MR. So I decided to put the answers to these inquiries here, on this platform, so that anyone with similar questions can benefit as well. There are tons of great articles out there already, but I hope to shed some light on some specific aspects of this tech that can sometimes be tricky to find answers to. Keep in mind, the questions below are from a designer’s perspective rather than a developer’s. But certain things could still apply to anyone. Enjoy!
Q&A
What skills are needed (or lacking) in the MR/AR industry that are not technical?
I believe this is a very important question. I see a lot of creative people wanting to dive into this industry but tend to forget they need to expand beyond a 2d mind-set to accommodate the new ways of thinking in AR/MR. Kind of like the mind-set one needs in training martial arts: Good to be anxious and eager to learn but you can’t just rush the process and overlook things. That would just spell disaster when it’s time to actually put what you learn to use, and in a project setting it wouldn’t just be bad for you but for your team members as well.
Besides new programs and workflows, you’ll have to divulge in the human factor throughout the process. So that would be understanding things like ergonomics, spatial awareness, embodied cognition (social and cognitive psychology), cognitive load, and proprioception when designing. There are a multitude of AR/MR resources online more than there ever has been dealing with each topics.
Also, the ability to be adaptable with whatever you need to get the job done and the knowledge, outside of just design principles, will be very essential. That flexibility and drive to learn is what will set you apart from the rest. Regardless of what you know at this point, self learning is the key and by practicing by doing, you can have a bright future in this field.
The best design work is never done by any one person alone, but instead is built from collaborative processes. Partnering with others in order to learn, evaluate, and make informed choices drives more impactful, effective solutions. — Jasmine Friedl
What school degrees complement UI/UX in MR/AR jobs?
To be honest, even though this tech has been out for quite a while it just only start blooming in the last few years. So if you don’t already have a degree that is favorable for this tech then no need to worry. There are other avenues of achieving those skills since things are still pretty new especially for companies hiring. Speaking in terms for a designer, you want look more into mixed media type of major’s but still keeping a solid foundation of graphic design and type. Even though this is a good basis for designers, I recommend you should also take advantage of your free electives or free time if you have any. Like front-end development, game design, computer science, psychology, sociology, industrial design, architecture, etc. A lot of these courses have intro classes that are available to any majors and I highly recommend getting a dose of any of it for a good foundation. So yes, you may be the master of none but you’ll gain something greater, knowledge that will elevate you to be the designer that will emerge from this XR field. Rather than being limited as 2D visual designer, now you’ll be able to contribute to your team in a wider range possible and have a greater increase in influence then before.
So if you're looking for a college to compliment the UI/UX. There are a lot university’s popping up with new curriculum everyday on the subject. Definitely a school with a combination of science and art (tech schools) I believe works best. I myself majored at RIT in New Media Design & Imaging which consist of motion graphics, 3d, code, graphic design, and animation so I’m a bit biased in this this as a first choice. If your current program does not have all the classes you need, talk to your department head and be open about what your interested in. You never know until you ask.
(below aren't in any special order)
- Rochester Institute of Technology
- Ringling College
- 2018 Best Colleges for Information Technology
- University of Toronto
If you don’t have the time or money to go back to school then no need to worry, there are plenty of online free and paid tutorials (cheaper than school). That is the great thing about being in the designer field because everyone will look at your portfolio and experience more than your degree. Even if your currently have a 2d designer background, You can show that you can juggle multiple mediums if you create side projects or experiments showing your understanding. Below are just a few you can start with.
- 3D for designers ( this is for c4d)
- 3Dsmax tutorials
- Unity 2D Creation
- Lynda.com
- Coursera
- VR dev school ( yea VR, but will learning all mediums will be helpful)
If needed, what would be an “absolutely must know” technical skill for MR/AR. What do you reckon would be the faster/best way to get it?
Coming down to technical skills, you have to know without a doubt 3D and game engines. It would be great to have a general understanding of backend and webgl flow, but as a designer you must have spatial awareness. There is no getting around that fact. Don’t fret if you don’t understand spatial AR/VR environments now, that’s why you must practice it. The learning curve is the new knowledge or programs and tech involved. There are no easy roads. But by doing and praciting through tutorials, You may pick it up quickly. Might take you a bit longer but you will make amazing things along the way. I would advise to focus on one thing at a time. Through online courses plan a roadmap for yourself for the next 6 months or so. I would start with learning a 3D program and or motion graphics. Blender is free to get a feel of it but if you really want to start off the bat right you might as well start with 3Dsmax or Maya. Not only understanding z-axis and spatial environments but the 3D lingo as well. Such as the terms polygons, object count, texture size, transparency, complex shaders, dynamic lighting and shadows and so on.
After 3D then I would dive into either Unity or Unreal. Unity is the most popular but is not great with more heavy/realistic 3D environments like Unreal. On the other hand, Unity is better with 2d mobile games and has more documentation than Unreal. Despite that, those aren’t the only two game engines that exist. There are of array of game engines out that, that can just as easily work but it will limit you in other areas such as a smaller range of developers that can work in those engines.
Of course this is a huge undertaking and not easy thing to learn but look at it more from a perspective like someone watching a movie rather than filming the actually movie. You don’t need to know it like the back of your hand but you should at least understand it and the lingo that is integrated in it’s interface.
Is there a standard best practice for UX/UI in MR/AR?
Just this question alone can produce a series of articles and debates. In the interest of time, I will have to be brief. There are currently tons of great creative people and resources defining the field of UX/UI within AR/MR lately, but it’s still actually very much the wild wild west.
Dealing with user experience, I feel there are less changes to that department compared to user interface standards. For UX, you can fall back on what you already know as a designer, this hasn’t really changed dramatically by the way you conduct the testing and research you need to execute. Yes the main platform has altered and adding additional hardware (haptics, controllers,etc) to consider depending on the device but the process for UX is adaptable. Along with including extra elements to consider like cues, prompts, microinteractions, gestures, audio, distance, motion sickness, orientation and so on.
I recommend using something I like to call adatable variables on when defining the project and before putting together the final workflow. This will determine what direction your workflow and timeline will be in. I will be brief here and just list the items for time sake:
- Game Style
- Installation Style
- Film Style
- Solo Experience
- Utility
Each requires a different experience and investment from the user. Adaptable variables can of course be interchangeable and combined. It can effect the team dynamics and workflow for that designated project.
Needing a place to start is very important in this day of age of immense information. So here are just a couple current articles to start with, but there are so many out there that are available online: (you can apply a lot principles to VR)
- Oculus — Best Practices
- UX of VR
- UX modes
- UX pointers for VR Designers
- The principles of good UX for augmented reality
If your new to UX though, I would take advantage on reading that up on that first before diving into the above:
A similar process to most AR/MR/VR user experiences is the UX in game development. Not identical but pretty close to things you’ll have to consider within the experience.
The best interface is no interface–Golden Krishna
Now concerning User Interface standards, a lot of designers are trying to confine it 2D standards, but it’s truly a fluid and undefined element. There are though some nice emerging articles (via Bushra) detailing the components needed for AR/MR. Of course having a foundation from 2D is not a bad thing, it’s good to still have experience and knowledge. Just know you’ll need to add to the knowledge and its evolving all the time as with any design practices. The chaos and beauty — the limitless boundaries and endless possibilities of this user interface is what had drawn me to it in the first place.
Despite that, there are a few standards I do see that fits well across and that you need to consider as you concept:
- Non-Diegetic: an overlay on top of the game, represented two dimensionally on the screen
- Diegetic: not on the screen, a physically object in the 3d scene to trigger an action or effect
- Meta: Can be coupled with Non-Diegetic UIs. For example in a shooting game, if you get shot the screen is overlay with blood to show that you are damaged.
- Spatial: exist three dimensionally. So the 2d UI is no longer stuck to the screen but within the spatial environment.
and concerning the relationship to your world or UI:
- Body Tag-lock: where the interface or an object will be locked to your orientation. So a 2d or spatial interface will be moving as you move.
- World Lock: (static/dynamic) you will be revolving around objects locked in the “world”. For example if you have an AR activated marker on a magazine cover, the 3d experience will be tied to its orientation. Letting you view around the object.
- Free-floating: not locked to anything. The user will have to physically turn it to view anything.
Also concerning VR interface, Mike Alger does have some nice talks about the methods involved in that: VR Interface Design Pre-Visualisation Methods.
What do you wish you knew about MR/AR before getting into the industry?
Well at first it wasnt getting a lot of public attention as it did now. So like many designers out there, I didnt even think about it as a possible career. Since we were so focused at the time with regular mobile. When I did finally get into it though, I wished I had really learned Unity. I did play with it years before (I like games) but only for a handful of times. Since I didn’t have any projects to apply it too, It was just satisfying my curiosity at the time.
Now I definitely recommend to take all the 2d tutorials you can and just absorb everything. There are tons of free assets you can play with as well, if you don’t know 3D at this point. I would look at Unreal as well but there is less documentation for that.
Anything you’d like to change about the industry (or wish was different)?
I’m a simple person, I had wished the term “Mixed Reality” was never created. The term mixed reality was subsequently coined to describe a hybrid of the two technologies. But let’s be real, mixed reality is really just augmented reality with a more focus on hand/body tracking. Maybe eventually though we will just say “XR” for everything.
Besides that, I know at a lot of designers wish there was a set of rigid design standards for XR but the lack thereof is actually the aspect I enjoy most. If you like conformity, then wrapping your head around XR practices might be more difficult long path ( but still attainable!) but I find current designers in this field thrive more in the beauty of it’s chaos. The more I think about it, maybe that’s how I really felt more driven in this field because I also had a subconscious dislike for rules and order for a platform that should be more fluid and adaptable.
Where would you say is the “MR/AR region” for jobs?
If you are a designer coming from a traditional mobile background, then you may have a harder time landing some roles. Since most companies (if they even post anything) are searching for more Unity/Unreal Designers than the 2D kind. Lately though it seems more companies are understanding that they do in fact need art directors, UI and UX to enhance the project. It’s kinda funny how I landed something in NYC, when that is probably the least optimal location for anything MR/AR/VR (company wise) but it’s changing steadily. Just all the big companies like google and facebook, have their main studios for those divisions on the west coast offices. Things are changing though, a lot of smaller studios and creative agencies all over trying to start their own divisions as well.
Most likely if the company is doing VR, they also might be looking into AR/MR. Also a very good source of jobs are in game studios as well.
West Coast
- Mountain View, CA
- Seattle
- menlo Park, CA
- San francisco, CA
- Los Angeles
East Coast
- Boston
- Raleigh, NC
- NYC
- Florida
Canada
- Vancouver, BC
- Toronto
- Montréal, QC
- Victoria, BC
Huge thanks to Jazna Rossi for her contributions in providing the questionnaire and review to this article. She is interested in making a move into the XR field, particularly with UX. Please reach out to her if you have any upcoming openings or internships.
If you have any further questions about anything in this field or design in general, message me on LinkedIn and I’ll write about it!
