A Retrospective of AI + Music
How AI has shaped music creation and the industry
The invention of electronic musical equipment in the mid-20th century revolutionized classical and popular music, and accelerated the development of new forms of music. As AI becomes more capable nowadays, machines play an increasingly important role in not only editing sounds, but also composing music. How has AI influenced music creation and music industry? Will AI take over musicians’ jobs?
History of AI and Music
First phase
At this time, research primarily focused on algorithmic composition that aims at an aesthetically satisfying new composition:
- In 1957, Lejaren Hiller and Leonard Isaacson from the University of Illinois at Urbana–Champaign programmed Illiac Suite for String Quartet, the first work completely written by artificial intelligence.
- In 1960, Russian researcher R.Kh.Zaripov published the first paper on algorithmic music composing using the “Ural-1” computer.
Breakthroughs
As research began to focus on understanding music, the more significant level of music intelligence emerged in generative modeling of music.
- In 1975, N. Rowe from the MIT Experimental Music Studio developed a system for intelligent music perception that enables a musician to play freely on an acoustic keyboard while the machine infers a meter, its tempo, and note durations.
- In 1980, David Cope from the University of California, Santa Cruz developed EMI (Experiments in Musical Intelligence). The system was based on generative models to analyze existing music and create new pieces based on them.
Current research
Research on AI and music continues on music composition, intelligent sound analysis, cognitive science and music, etc.
- Initiatives such as Google Magenta, Sony Flow Machines, IBM Watson Beat, want to find out if AI can compose compelling music.
- Musicians, such as Taryn Southern, have been collaborating with open source music composition software to create music.
What can AI and music do?
Current research and applications of AI and music covers 6 major categories: sound processing, music analysis, composition, performance, curation, and education.
Analysis
AI models can deal with source separation, music transcription, structure analysis, instrument recognition, emotion recognition, etc. Such models are the foundation of intelligent music applications. The application of AI in music analysis has been a primary focus of both academia and industry.
- Wekinator (An open-source software allowing anyone to use machine learning to build new musical instruments, gestural game controllers, computer vision or computer listening systems, and more)
- Google Magenta (An open-source research project exploring the role of machine learning as a tool in the creative process)
Composition
Also, AI can compose new music, either unconditioned (from random seeds) or conditioned (e.g. given a prime melody). Several software applications has been building models to help artists and enterprises to create music.
- Amper (enabling enterprise teams to compose custom music in seconds)
- AIVA (composing emotional soundtrack music and a creative assist for creative people)
- Flow Machines by Sony (augmenting music creativity)
Performance
With generative modeling, AI now also adds expressiveness to musical audios created from scores and brings the music to life.
- Performance RNN by Magenta (Generating music with expressive timing and dynamics)
- NotePerformer (Artificial Intelligence-based playback engine for musical notation)
Sound processing
AI models can also perform a wide variety of signal processing operations from automatically mastering music to creating new sounds.
Streaming, curation, and monetization
With the help of AI and neural networks, it is much easier to categorize music and find deeper similarities among music pieces. Music streaming services and musicians can distribute music to their target audiences more efficiently.
- Spotify (personalized music recommendations powered by AI)
- Musiio (powering tagging and searching for music, and curating playlists)
- LANDR (polishing, distributing, and promoting songs for musicians)
Education
A lot of musical applications have been using AI to help music enthusiasts improve by analyzing their playing and providing instant feedback.
- Yousician (an interactive music service to learn and play a musical instrument)
- Jamstik (Travel-size MIDI guitars with teaching and creating apps)
People + AI + Music + Business
As more and more artists begin to utilize AI to boost their music creation, there are some important implications for musicians and music business.
Is AI a threat to musicians?
The capabilities of AI create tension among the musicians and producers communities, who first view it as a threat to their jobs. The argument from the AI-music startups is that AI is a creative tool that frees musicians up to make more/better art.
Although AI can outperform humans in areas such as video backing music or soundtracks to help you sleep, it is not capable of generating great original music without human input.
“Currently, the best AI music is just listenable, albeit usually with human input when it’s being created.”
- Music Ally
How should AI assist musicians?
Similar to the levels of self-driving cars, we can level AI music from level 0 to 4. It is often up to the musicians what level of assistance they want from AI.
- Level 0: artists create music without any assistance from AI
- Level 1: artists use AI for sound processing but not composition
- Level 2: artists use AI to assist them compose music
- Level 3: the music is created by AI but performed by a human
- Level 4: the music is created and performed by AI
“Will AI assist the musician or is it going to be the musician? The answer appears to be both.”
What are the business models for AI + music?
Outside of research, there are several common business models for AI music applications:
- A soundtrack composer for small and medium-sized enterprises or consumers (video and game creators, social media users)
- A compositional tool sold to musicians and producers
- Hardware or software that helps music enthusiasts learn music
- Services that distribute and promote music
Licenses and copyrights for AI-composed music?
If an AI model is trained on the entire backlog of an artist. Would that artist have any claim on songs that come out of that AI and the revenues generated by those songs?
At the same time, would AI-composed music be protected by copyright laws even if they are based on music pieces of other artists?
“Most jurisdictions, including Spain and Germany, state that only works created by a human can be protected by copyright. But with the latest types of artificial intelligence, the computer program is no longer a tool; it actually makes many of the decisions involved in the creative process without human intervention.”
Final thoughts
As machines and AI become more powerful and accessible, AI will inevitable transform the music industry. However, the choice is on us how we work with AI. I like the quote from YACHT band during Google I/O:
“It’s not so much about, as an artist, being replaced by machine learning, but rather being given the opportunity to focus our energies in different directions and different places we are accustomed to. It’s not about revoking control, but holding on and let the process change you.”
Will AI take over musicians’ jobs? Maybe some of them. The reason why robots can take over our jobs is because we make the jobs robotic. But music creation and production is a creative process. It is important to give musicians more control over AI applications instead of letting AI take over. After all, we love musicians for their humanity and personality — not just the music itself.
