A Primer for Computer Graphics in VR/AR Design
Why does Every VRAR Practitioner Need to Know Computer Graphics?
Every digital consumer actually can use Computer Graphics(CG) a bit, but VRAR practitioners need to grasp CG a little bit more.
We’re living in a world surrounded by computer-generated content. Whether you’ve noticed or not, we’re dealing with a lot of concepts originating from Computer Graphics(CG) in our daily lives. It happens when you’re using your mobile phone to polish a digital photo by adjusting its size, color, and brightness to make the photo social-media ready. It happens when you’re purchasing a laptop and checking the GPU specs to make sure it is powerful enough to run the games you love to play. If you’re a digital game player, Frame Rate or FPR (frames per second), Render, etc. — those CG words might be too familiar to you to bother searching them up. As a digital consumer, you might not know what CG is, technically, but it doesn’t affect you to use it practically.
But as a digital content creator, the nature of the work requires a little bit more understandings of Computer Graphics. But luckily, for most of the job titles like illustrator, animator, UI designer, or game developer, as long as you have proficiency in using job-specific tools like Photoshop, 3D modeling tools, or game engine, you could survive with a small amount of superficial knowledge related to CG.
That’s not the case for VRAR content creators though. Publishing a consumer-ready VRAR application is far more challenging than making a popular mobile app. The CG knowledge gained from flat-screen experiences is just the tip of the overall CG iceberg.
For a VRAR practitioner, if you want to create high-quality VRAR content, you’ll need to study Computer Graphics more comprehensively and systematically.
Why VR/AR apps are harder to create than regular apps?
📸🎥📱💻🖥
Consumers of VRAR products have high expectations. Our digital-age consumers are spoiled by mobile apps with fancy features, sci-fi movies with marvelous visual effects, and immersive games with photorealistic world design. We are picky when it comes to visual products generated by computers. Emerging technologies like VRAR are taking all those pickiness a step further. People expect photorealistic world design and physics-realistic interaction to do something that can’t be done by any other digital application, like teleport yourself to any place on earth with Google Map.
Technically, creating a VRAR product is more challenging than its digital peers. On the surface, VRAR products are simply putting two small pieces of displays full of pixels in front of eyes to trick the viewer that this is real. Seemingly identical to its digital peers, VRAR is also made out of pixels. However, given the amount of calculation required by a VRAR app to fill each pixel in colors, VRAR products raise higher requirements for computing power and quality compared to any traditional digital medium.
Unlike mobile apps that consist of 2d images, VRAR apps are full of three-dimensional shapes. Those shapes will exist inside a virtual world where digital photons bounce off between those virtual surfaces and change their color when hitting them. Compared to movies with pre-rendered visual effects, VRAR expects the computer to generate specific effects in real-time response to players’ behaviors and commands. This is similar to 3D games running on our PC. But PC games normally limit players’ view within a small corner of the game world, while VRAR apps open up the world’s 360° view for users to explore. Additionally, VRAR allows us to interact with the computer-generated content not merely by seeing but also by moving our bodies, talking, touching, hearing, even smelling.
90 Frames Per Second
90 is the standard fps (Frames per second (fps)) for running a smooth VR experience. The amount of data that a VRAR-ready chip needs to handle in one millisecond during the runtime is massive.
Frame Rate Explanation
VRAR becomes the hottest battlefield for CG professionals, with no doubt.
I’m not saying that we need to become CG professionals in order to work in this field. But challenging problems do raise the bar for people who want to work in the VRAR Industry. My experiences as a VRAR practitioner for six years tell me that, in the VRAR talent market, people with multidiscipline backgrounds that have a good understanding of computers, graphics, or computer graphics are high-demands. In my opinion, for VRAR practitioners who want to competitively work in this field, don’t hesitate to spend time and effort to learn more about Computer Graphics and the graphics pipeline.
How much CG knowledge can make you competitive in the VR/AR industry?
Let me first tell you how much, in my opinion, is not enough.
🎨🌞 📐📈📱 🎮 🎬 🖥 ≠ CG👩🏫
I used to think I knew a lot about CG. For the Graphics part, I have gone to a weekend art school for eight years to get systematic training in sketching and painting and then studied Industrial Design and Engineering at college. For the theoretical CG part, when I was at college, I took a course called Computer Graphics Basic, which inserted most of the CG keywords into my mind computer. Vector Math, Linear Algebra, and Optical Physics, which are the foundational subjects to Computer Graphics, are not blockers for me to grasp. Practically, I have rich experiences of creating mobile apps, websites, or games and being skilled in using Photoshop, Rhino for modeling, and Unity for game development.
🤓 Yes, I’m a nerd. So I thought it was fair to conclude that I have a good understanding of Computer Graphics, which should prepare me well for creating VRAR (immersive) products.
💔 But I was wrong. The illusion was broken until the first time I worked as a UI designer for a VR application. I found myself barely getting the color of a 3D menu aligned with the color I pictured in my mind. My intuition for this problem was to adjust the material’s color attached to that menu model, but it didn’t work here. Then I tried to change the shader of the material from system Standard to a UI-specific one. This move made the menu color correct but aesthetically fake, which was not what I desired. When I finally achieved the visually accurate result, once I started to run the game and moved the menu around with my controllers, the color looked different. Then I realized that as the point of view changes or the environmental lighting condition varies, the color of objects would also get affected accordingly.
🙉 I got stuck. Time spent, efforts paid, but what I produced was still not fine-tuned with the effect I wanted to achieve. Color-filling used to be a simple task when I was a mobile app UI designer. What I need to do is simply picking the right color from a color wheel, maybe plus minor color adjustments for different display setups afterward, but that’s it. Such work can be done perfectly with the help of image processing editors like Photoshop. But when it comes to VR/AR apps, a simple color modification might need complex setups to achieve.
The process of creating a VRAR product is full of uncertainties. Color issues primarily relate to the software level of the graphics pipeline, but more critical issues sometimes come all the way down to the hardware and user level. When I worked as an interactive engineer for another immersive app, I realized that perfect codes showing no error didn’t always guarantee a smooth interaction result. Tons of variables — hardware setup, GPU and CPU models, headset connection, wifi speed, and physical room lighting — will swing the result of one input. Accumulated Calculation delays caused by one pipeline bottleneck can lead to customer loss, which is not desirable.
Due to the complexity of the graphic pipeline, issues of a VRAR app are easy to spot, but the root causes are hard to locate.
Don’t know what causes the problem, can’t achieve the effect you want, can’t predict the output, lost sense of what is possible to generate and what is not technically ready — those are the symptoms I identified as not reaching the baseline of becoming a qualified VRAR content creator. And our experiences in the traditional 2d domain are very likely to blind us from noticing them.
To become a good VRAR content creator, familiarities with CG key concepts or proficiency in a modeling or image processing tool or a game engine are just the starting point. From here, in-depth knowledge of the VRAR graphic pipeline — from the physical room setup to the user biological condition and from the hardware to the software — is the key to beat the uncertainties of what we produced and control the output of the pipeline.
A simple way to test your understanding of Computer Graphics is by asking yourself the following questions:
Can you explain how each pixel on the screens of a VRAR headset gets colored during the runtime?
Can you explain how two-pixel images trick the user into thinking the virtual world is real?
If the answers to those questions are NOs, you need to take Computer Graphics more seriously.
How can I get started to learn more about CG for my VRAR projects?
🤯 Emmmmm. It seems like I have a lot to learn … a lot of Math, Code, Weird Terms.
🍝 CG does appear too difficult to learn from scratch. Check out the Wikipedia page for “Computer Graphics.” You might get panic. Tons of related concepts and models are overwhelming for a CG newbie to pull one of the threads to start. If you search “how to learn CG” on Google, you might get advice to learn vector match or linear algebra first. It is misguiding.
Computer Graphics, to me, is a field that studies how to train computers to learn how to draw and what to draw.
It is never a tedious and boring subject that only about math, computer, and algorithms. CG is a multidiscipline subject related to many domains, like:
Human: Vision and Cognition
Physics: Optics and Waves
Computer: Hardware and Software
HCI(Human Computer Interaction)
Vector Math & Linear Algebra
Graphics: Drawing Techniques
It sounds complex, but CG has its roots in how we as humans perceive this visual-dominant physical world.
To set up a study plan, you need to ask yourselves three questions: what to learn? and how to learn? and what I want to achieve?
① What to learn?
Pick your learning focus based on your strengths and weaknesses. Before learning, a correct evaluation of your knowledge structure is necessary to determine what to focus on. If you know how to draw, then you can quickly grasp how the computer shades a mesh. Optics physics knowledge— how photons bounce off different types of surfaces to create this colorful world — can help you comprehend diverse reflection models and rendering methods. Understandings of how human eyes see objects can connect you to the underlying reasons why some hardware or algorithms design in that way, such as why colors of pixels are represented by RGB(Red Green Blue) and why the virtual camera of a VR app should set the field of view angles equal to 60.
What to focus on also depends on the roles you want to play in a VRAR project; you can approach CG from different angles and levels of detail.
As a VRAR product owner, you might want to focus on technology trends and hardware capabilities, like which headset is budget-friendly and system compatible for your target users.
As a VRAR visual designer or digital artist, to get the picture out of your mind and digitalized it accurately, you should have a comprehensive grasp of the project graphic setting before diving into the nuances of the visual details. Rendering pipeline, Model Mesh, Environment lighting, Texture, Materials, Shaders are the key knowledge points you need to map off in your heart.
As a VRAR engineer, you should always be cautious of two issues: simulate the Reality or create an alternative, and quality or performance.
② How to learn?
My recommendation to learn CG is using a project-based approach： Learning by doing. Theory plus practice.
Rome was not built in a day. Likewise, learning CG has no shortcuts and no endpoint. It is a journey full of trials and errors but also fun. So if you feel interested in my journey of learning Computer Graphics for VRAR projects, please follow me. In the next article, I would like to share the following things with you:
How do I learn Computer Graphics for creating good VRAR content? and What online materials are awesome for CG newbies.
How can you achieve such understanding without a technical background or a CG college degree?
Which part of CG knowledge helps me most to tackle specific problems relates to various job titles I have experienced for different VRAR projects( HCI researcher, UXUI designer, digital artist, and interactive engineer)?
Thanks for reading!
Hi, I’m a VR/ AR UX Engineer, been in this industry for six years as a designer, researcher, and developer. Feel free to leave your valuable thoughts here.
If you are also interested in VRARMR, let’s connect at
My Linkedin: https://www.linkedin.com/in/yu-qin-01a41212a/
Twitter: https://twitter.com/EllenYQin
My Portfolio:https://www.qindayu.com/ (under repair)
Glad to meet you here！
