Member-only story
Behind the Magic: How we built the ARKit Sudoku Solver
What we learned from our first foray into Machine Learning
Brad Dwyer is the founder of Hatchlings, a startup that makes games and mobile apps in Des Moines, Iowa.
A few weeks ago my company, Hatchlings, released Magic Sudoku for iOS11. It’s an app that solves sudoku puzzles using a combination of Computer Vision, Machine Learning, and Augmented Reality. You can try the app by downloading it from the App Store on your iPhone 6S (or newer) or iPad Pro.
We have gotten a ton of attention including getting voted to the front of 9GAG, being listed as a top ARkit app by places like Gizmodo and CNET, and being called out by Fast Company’s design blog as one of the only AR apps that doesn’t “stink”.
Many people have asked me about the app so I thought it would be fun to share some behind the scenes of how and why we built it.
This is the second post in a 3-part series. Part one explained why we built Magic Sudoku and what makes it different from other augmented reality apps, and part three will explore the business side.
Subscribe or follow me on twitter and you’ll be the first to know when part three comes out. And don’t forget to download the app and give it a try yourself!
How does Magic Sudoku work?
Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. -Steve Jobs
Magic Sudoku uses a combination of Computer Vision, Machine Learning, and Augmented Reality to give a magical user experience that “just works” when you point your phone at a Sudoku puzzle. The basic flow of the application is this:
- ARKit gets a new frame from the camera.
- We use iOS11’s Vision Library to detect rectangles in the image.
- If rectangles are found we determine whether they are a Sudoku.
- If we find a puzzle we split it up into 81 square images.
- Each square is run through a neural network we trained to determine what number (if any) it represents.
- Once enough numbers are gathered we use a traditional recursive…
