AI Can Help Anyone Become a Beatbox Champion

To help up-and-coming musicians create the best beats for their song, developers from a Japanese-based AI startup developed a deep learning system called Neural Beatboxer that can convert everyday sounds into hours of automatically compiled rhythms.

Users can visit their website, feed it some sounds, and the neural network automatically produces a custom drum kit that can go on for hours.

Nao Tokui, CEO of Qosmo, said he came up with the idea while working as a DJ in Japan. He feels frustrated with the current direction of AI music generation research and hopes his neural network will help.

Using NVIDIA Quadro GP100 GPUs, with the cuDNN-accelerated Keras deep learning framework, Tokui built a convolutional neural network audio classification model.

“I used a drum machine sound dataset available online and trained a convolutional neural network to classify audio based on its spectrogram,” he explained. “The model was trained to classify the following drum sounds: kick, snare, hi-hat-closed, hi-hat-open, low tom, mid tom, high tom, clap, and rim.”

For the rhythm generation, he used a pre-trained drums recurrent neural network provided by Google’s Magenta project.

The web front-end is built with TensorFlow.js, magenta.js, and p5.js.

“Initially, I was thinking of using the same technique to make remixes of music I play during my DJ set,” Tokui explained. “I’ve been working on a project called AI DJ, where I play music alongside with an AI DJ back to back.”

He says it would be great if an AI DJ could select and mix music, and also remix it in a way that humans can’t in real-time.

“My intention here is to make interesting/weird/eccentric novel music/rhythms,” Tokui said.   “Using raw sound materials recorded via microphone, this system may be able to generate exciting/novel rhythms, which no one cannot (or doesn’t want to) compose manually by himself without any help of the AI,” he explained.

Try it >