Integrating Super Resolution and Low-bitrate Codec Language Model for High-Fidelity Text-guided Music Generation
Abstract We introduce InspireMusic, a unified framework that integrates super resolution and a language model for text-guided generation of high-fidelity long-form higher sampling rate music. Different from previous approaches, InspireMusic utilizes a low-bitrate single-codebook audio tokenizer that achieves high reconstruction performance while containing richer semantic information than competitive neural acoustic codecs, thereby enhancing the efficiency of training and inference. An autoregressive transformer large language model (LLM) based on Qwen2.5 is used to predict audio tokens. Through incorporating the audio tokenizer, the language model, and the super-resolution flow-matching model, InspireMusic can generate high-quality and coherent long-form music. We propose a distribution-aware dual-temperature truncation scheme for robust token selection and pair it with a positive-momentum projection guidance mechanism that keeps the autoregressive decoder stable during music generation. Comprehensive experiments demonstrate that InspireMusic achieves comparable to superior performance compared to top-tier open-source and closed-source music generation systems, in both objective and subjective evaluations.
Contents
- Music Generation Model: InspireMusic
Overview of InspireMusic
Figure 1. Overview of InspireMusic, which is composed of an audio tokenizer, an autoregressive transformer LM, and a super-resolution flow-matching model. (1) Lower-sampling-rate audio is converted to discrete tokens via an ultra-low-bitrate audio tokenizer. (2) The audio and text tokens are fed to an autoregressive LLM to generate tokens based on next token prediction. (3) The flow-matching model maps the generated tokens to the latent features with high-resolution fine-grained acoustic details obtained via HiFi-Codec from higher-sampling-rate audio, to ensure the acoustic information flow connected with high fidelity through models. (4) The vocoder decoder produces high-quality 48kHz audio waveform.
Text-to-Music Generation
This section showcases a collection of musical pieces generated by InspireMusic.
The music was generated by InspireMusic and MusicGen, serving as a comparison of short-form text-to-music generation.
| Text Description | InspireMusic-1.5B-Long w/o SRFM (1.5B) | InspireMusic-Base w/o SRFM (0.5B) | MusicGen-Small (300M) | MusicGen-Medium (1.5B) | MusicGen-Large (3.3B) | ||
|---|---|---|---|---|---|---|---|
| Experience soothing and sensual instrumental jazz with a touch of Bossa Nova, perfect for a relaxing restaurant or spa ambiance. |
Music Generation w/ Different Sampling Methods
This section showcases a collection of musical pieces generated by InspireMusic, utilizing Dual-Temperature Consistency Truncation (DTCT) sampling, repetition-aware (RAS) sampling and Top-k sampling methods, respectively.
| Text Description | Sampling Method | InspireMusic w/ SRFM | InspireMusic w/o SRFM |
|---|---|---|---|
| Experience soothing and sensual instrumental jazz with a touch of Bossa Nova, perfect for a relaxing restaurant or spa ambiance. | DTCT | ||
| RAS | |||
| TopK=350 |
Music Generation with & without Super Resolution Flow Matching
This section showcases a collection of musical pieces generated by InspireMusic with and without Super Resolution Flow Matching (SRFM), utilizing DTCT sampling.| Music Structure | Genre | Text Description | Audio Output w/o Super Resolution Flow Matching | |
|---|---|---|---|---|
| Intro | Instrumental | A delightful collection of classical keyboard music, purely instrumental, exuding a timeless and elegant charm. | ||
| Chorus | R&B | A soothing blend of instrumental and R&B rhythms, featuring serene and calming melodies. | ||
| Verse | Jazz | Experience soothing and sensual instrumental jazz with a touch of Bossa Nova, perfect for a relaxing restaurant or spa ambiance. | ||
| Outro | Rock | The instrumental rock piece features dynamic oscillations and wave-like progressions, creating an immersive and energetic atmosphere. The music is purely instrumental, with no vocals, and it blends elements of rock and post-rock for a powerful and evocative experience. |
Long-form Music Generation
| Generated Audio Length | Text Prompt | InspireMusic-1.5B-Long |
|---|---|---|
| 1min | The instrumental rock piece features dynamic oscillations and wave-like progressions, creating an immersive and energetic atmosphere. The music is purely instrumental, with no vocals, and it blends elements of rock and post-rock for a powerful and evocative experience. | |
| The instrumental piece blends smooth jazz with an acid jazz twist, creating a serene and sophisticated atmosphere. | ||
| Experience soothing and sensual instrumental jazz with a touch of Bossa Nova, perfect for a relaxing restaurant or spa ambiance. | ||
| A serene blend of instrumental and light pop, featuring soothing melodies and a gentle, soulful keyboard performance. The music likely includes Chinese vocals, possibly delivered by a male singer, enhancing the tranquil and reflective atmosphere. | ||
| A soulful blend of instrumental blues and jazz, featuring a serene and atmospheric soundscape. | ||
| A soothing blend of instrumental and R&B rhythms, featuring serene and calming melodies. | ||
| Experience a captivating blend of classical opera and concert arias, featuring expressive Italian vocals and a rich orchestral backdrop. The performance showcases a powerful female voice, delivering the timeless elegance and emotional depth characteristic of Mozart's compositions. | ||
| A soothing instrumental piece blending elements of light music and pop, featuring a gentle guitar rendition. The overall feel is serene and reflective, likely instrumental with no vocals. | ||
| Embracing a serene and electronic ambiance, this compilation features instrumental tracks with a slow tempo, evoking a tranquil and modern digital soundscape. | 3min | A collection of soothing instrumental and contemporary pop tracks, evoking a sense of modern trends. |
| The soothing piano melodies evoke a serene and contemplative atmosphere, blending classical and contemporary styles. | ||
| A vibrant orchestral piece with a classical baroque feel. | ||
| Experience classical music with light smooth rhythm. | ||
| Experience soothing instrumental jazz with a peaceful piano backdrop, perfect for relaxing evenings. | ||
| A serene blend of instrumental and piano melodies designed for relaxation, meditation, and contemplation, featuring elements of New Age and healing music. | 5min | The instrumental piece blends smooth jazz with an acid jazz twist, creating a serene and sophisticated atmosphere. |
| The instrumental rap track exudes a classic boom bap vibe, characterized by its French hip-hop roots and a smooth, rhythmic flow. | ||
| A soulful blend of instrumental blues and jazz, featuring a serene and atmospheric soundscape. | ||
| A delightful collection of classical keyboard music, purely instrumental, exuding a timeless and elegant charm. | ||
| A soothing collection of instrumental and classical piano pieces, blending light and pure music with a touch of popular instrumental styles. | 8min | A delightful collection of classical keyboard music, purely instrumental, exuding a timeless and elegant charm. |
| A serene blend of instrumental and piano melodies designed for relaxation, meditation, and contemplation, featuring elements of New Age and healing music. | ||
| The instrumental rock piece features a prominent bass guitar, delivering a pure and energetic sound. | ||
| The classical instrumental piece features a lively and intricate violin performance, blending elements of orchestral and light music with a serene and flowing feel. |
Music Continuation
This section showcases a collection of musical pieces generated by InspireMusic for music continuation task.
Music Continuation Samples
This section showcases a collection of musical pieces generated by InspireMusic for music continuation task, audio output with Super Resolution Flow Matching.| Music Prompt | InspireMusic-1.5B-Long | InspireMusic-1.5B | InspireMusic-1.5B-24kHz | InspireMusic-Base | InspireMusic-Base-24kHz |
|---|---|---|---|---|---|
| Music Prompt | InspireMusic-1.5B-Long_no_flow | InspireMusic-1.5B_no_flow | InspireMusic-Base_no_flow |
|---|---|---|---|
More Diverse Generation Samples
This section showcases a collection of musical pieces generated by InspireMusic with different music structures and genre.InspireMusic w/ DTCT Sampling
This section showcases a collection of musical pieces generated by InspireMusic for short-form text-to-music generation with super resolution flow matching, utilizing DTCT sampling.
| Text Description | Audio Output |
|---|---|
| Generate a gentle classical piano music piece tune featuring a romance night atmosphere. | |
| The soothing piano melodies evoke a serene and contemplative atmosphere, blending classical and contemporary styles. | |
| Experience a dynamic and passionate classical concerto, characterized by its energetic and fiery movements. This instrumental piece, performed by a symphony orchestra, showcases the intricate and expressive style of Romantic-era music. | |
| A soulful blend of instrumental blues and jazz, featuring a serene and atmospheric soundscape. | |
| A serene blend of instrumental and piano melodies designed for relaxation, meditation, and contemplation, featuring elements of New Age and healing music. | Experience classical music with light smooth rhythm. |
InspireMusic w/ Repetition-Aware Sampling
This section showcases a collection of musical pieces generated by InspireMusic for short-form text-to-music generation with conditional flow matching, utilizing repetition-aware sampling.
| Music Structure | Genre | Text Description | Audio Output |
|---|---|---|---|
| Intro | Instrumental | A serene instrumental piece with a nostalgic feel, blending traditional and modern elements for a soothing auditory experience. | |
| Metal | Exploring a blend of symphonic metal with a primal, rhythmic instrumental backdrop, this track features powerful, creating an epic and immersive auditory experience. | ||
| Pop | Featuring a soothing blend of Mandarin pop with a gentle instrumental backdrop, this track offers a serene and reflective ambiance. | ||
| Instrumental | A serene blend of instrumental modern classical music, evoking a contemplative and tranquil atmosphere. | ||
| Electronic | Experience an electrifying journey through high-energy electronic beats and mesmerizing trance rhythms. | ||
| Game | The instrumental track from a video game soundtrack exudes an adventurous and lively atmosphere, characterized by its energetic and playful melodies. | ||
| Verse | Hip-hop | Laid-back | The track exudes a laid-back, instrumental hip-hop vibe with a relaxing beat. | |
| Piano | A serene and elegant classical piano piece, evoking a sense of tranquility and nobility, with a purely instrumental arrangement. | ||
| Game | The instrumental and anime-inspired music evokes a nostalgic journey through classic video game soundscapes, blending energetic and atmospheric elements. | ||
| Latin | A vibrant blend of Latin and Latin American instrumental music, likely featuring in a heartfelt and rhythmic style. | ||
| Electronic | The electronic track exudes a warm, wintery vibe with instrumental. | ||
| Electronic | The track delivers an immersive electronic experience with a hypnotic, psychedelic atmosphere. | ||
| Electronic | The track delivers an immersive electronic experience with a hypnotic, psychedelic atmosphere, featuring English vocals that add a captivating layer to the progressive soundscape. | ||
| Punk | The music exudes a gritty, rebellious energy with a raw punk vibe. | ||
| Chorus | Classical Instrumental | A captivating piece of classical instrumental music, characterized by its intricate variations and timeless elegance. | |
| Electronic, Folk | The instrumental track blends electronic and folk elements, creating a serene and atmospheric soundscape. | ||
| Reggae | Experience a deep, instrumental journey through reggae and dub, characterized by echoing rhythms and a laid-back, atmospheric vibe. The music is purely instrumental, allowing the rich, resonant basslines and intricate soundscapes to take center stage. | ||
| Electronic | TThe music exudes a serene and atmospheric electronic vibe. | ||
| Instrumental | Playful | Whimsical | The instrumental piece exudes a playful and whimsical atmosphere, likely featuring lively and rhythmic elements. The music seems to be inspired by nature and animals, creating an engaging and light-hearted experience. | ||
| Electronic | An energetic electronic track with a trance-like feel, featuring instrumental melodies and a dynamic beat. The music likely has a Chinese influence, given the tags, and is purely instrumental without any vocal performance. | ||
| Electronic | A soothing blend of instrumental and electronic sounds, likely featuring relaxing and ambient tones. The music is instrumental, suggesting no vocals, and it evokes a serene, space-themed atmosphere. | ||
| Instrumental | The instrumental track exudes a serene and reflective ambiance, characterized by its smooth and melodic composition. The absence of vocals allows the music to create a tranquil and contemplative atmosphere, perfect for relaxation or introspection. | ||
| Jazz | A serene blend of contemporary jazz and instrumental music, characterized by drifting melodies and a smooth, floating ambiance. The piece likely features instrumental performances without vocals, creating a tranquil and sophisticated atmosphere. | ||
| Reggae | Pop | A vibrant blend of reggae and pop, delivering a soulful and energetic performance. | ||
| Outro | World Fusion Music | Exploring the vibrant rhythms and intricate beats of Egyptian percussion, this offers a captivating journey into world fusion music, blending traditional sounds with modern influences. The vocals, likely in Arabic, add an authentic touch, with a dynamic performance that complements the energetic and immersive instrumental backdrop. | |
| Electronic | An energetic electronic track with a trance vibe, featuring instrumental elements and a dynamic beat. | ||
| Electronic | The instrumental track exudes a progressive house vibe with electronic elements, creating a serene and futuristic atmosphere. | ||
| Rock | Funky | The instrumental rock track exudes a funky, energetic vibe with a touch of rock elements, creating a dynamic and engaging listening experience. | ||
| Electronic | The instrumental track exudes a serene and atmospheric electronic vibe, blending ambient sounds with a touch of tranquility. |
InspireMusic w/ TopK Sampling
This section presents the demo of shortform instrumental music generation with 30 seconds pre-trained InspireMusic-Base model based on Top-K sampling method.
| Music Structure | Genre | Text Description | Audio Output |
|---|---|---|---|
| Verse | Pop | Dreamy and ethereal, this English-language track blends elements of pop and dream pop, featuring a remix that adds a neon-lit twist. The vocals, likely delivered by a male singer, float effortlessly over the lush, atmospheric soundscape. | |
| Pop | A lively and rhythmic instrumental track, characterized by its upbeat and danceable pop style. | ||
| Classical | A serene instrumental piece blending classical and contemporary elements, performed in English, evoking a timeless and elegant atmosphere. | ||
| Electronic | A vibrant electronic instrumental experience blending drum and bass elements, likely featuring soothing Chinese vocals. | ||
| Rock | A dynamic blend of instrumental rock and math rock, featuring intricate rhythms and a powerful, energetic feel. The music is purely instrumental, allowing listeners to immerse themselves in the complex, yet harmonious soundscapes. | ||
| Electronic House | Dance | The track exudes a captivating electronic house vibe, characterized by its instrumental purity and energetic dance beats. | ||
| New Age Music | Experience soothing instrumental New Age music designed to balance chakras, enhance happiness, and maintain calm, perfect for yoga sessions and relaxation. | ||
| Game Music | An energetic blend of Japanese anime and game music, featuring dynamic instrumentals and possibly female vocals, creating a nostalgic and vibrant atmosphere. | ||
| Chorus | R&B | The music exudes a soulful and rhythmic vibe, blending elements of R&B and soul. The performance likely features a smooth, emotive delivery. | |
| Electronic | Anime | A mysterious and atmospheric track, blending elements of dark electronic and anime soundtrack styles. | ||
| Jazz | Imagine a lively instrumental jazz piece with a swinging rhythm that evokes a sense of joy and nostalgia. The music, characterized by its upbeat tempo and smooth melodies, creates an atmosphere perfect for dancing or simply enjoying a cheerful moment. | ||
| Electronic Dance | The energetic electronic dance track features nostalgic vibes. | ||
| Electronic | An energetic electronic track with a trance-like feel, featuring instrumental melodies and a dynamic beat. The music likely has a Chinese influence, given the tags, and is purely instrumental without any vocal performance. | ||
| Instrumental | A serene and melodic Chinese ballad featuring gentle, harmonious with an instrumental version that highlights traditional musical elements. | ||
| Heavy Metal | The intense live performance delivers a raw and aggressive sound, characterized by heavy, distorted guitars and powerful drumming, delivered with a harsh, guttural style, typical of extreme metal genres. | ||
| Energetic | Cardiac Sound | The instrumental track delivers an energetic and rhythmic experience, characterized by a pulsating beat and dynamic soundscapes. | ||
| Indian Classical | A soothing blend of traditional Indian classical music featuring intricate flute melodies and rhythmic tabla beats. | ||
| Intro | Electronic Dance | A serene instrumental piece with a nostalgic feel, likely featuring male vocals in Mandarin, blending traditional and modern elements for a soothing auditory experience. | |
| Electronic | The track exudes a deeply immersive electronic vibe, characterized by pure instrumental sounds and a rich deep house rhythm. The overall feel is intoxicating and hypnotic. | ||
| Outro | Modern Instrumentation | A soothing blend of traditional Chinese melodies and modern instrumentation, featuring gentle that evoke a serene and tranquil atmosphere. | |
| House | The vibrant mix of house and classical crossover creates an energetic and sophisticated atmosphere, featuring with a powerful performance. | ||
| Hip-hop | Rap style | A dynamic blend of West Coast hip-hop with a storytelling rap style, featuring English vocals that convey a gritty, urban narrative. | ||
| Latin dance | The vibrant and energetic rhythms of this music blend Latin dance styles with a global influence, featuring lively salsa and merengue beats. The vocals, likely in Spanish, are delivered with passionate intensity, possibly by a male singer. |
Disclaimer
The content provided above is for academic purposes only and is intended to demonstrate technical capabilities. Some examples are sourced from the internet. If any content infringes on your rights, please contact us to request its removal.