Translation has never been what the sci-fi shows promised us. Far from instant, accurate language exchange, creating video voiceovers in multiple languages usually means finding fluent speakers and paying a premium for their expertise. AI video translation offers another way. But does it work?
IdeaRocket Founder and Creative Director William Gadea thinks so. “I did a video of me speaking in English and it translated it to sound just like me speaking in Spanish. The team couldn’t tell the difference,” he said.
That’s high praise, considering many of our team members are native Spanish speakers who work with Will on a daily basis. Here’s a closer look at the current state of AI video translation and how you can use it for your explainer videos and commercials.
HeyGen AI Video Translation
The translation tool that so impressed Will and our team was HeyGen AI Voice Translator. It promises to translate your videos into more than 70 languages and 175 dialects while maintaining tone and delivery. There’s an AI dubbing version you can use for animated video or voiceover. But it doesn’t just generate voice overs. If your speaker is visible on screen, HeyGen actually clones your voice and lip-syncs your video so you can use one voice actor (or yourself) to create multiple translated videos.
The demo below, shared by HeyGen, showcases not just translation, but also more subtle nuances of accent and regionalism. (Note that this video is about a year old, so the information in the second half of the is outdated.)
The interface is simple. Just click the button to start a translation project, upload or drag and drop your video, and choose a target language. You can also load a video from a YouTube or a Google Drive link. The free plan limits you to a duration of 3 minutes or less, but paid plans (starting at $29 per month) can handle videos up to 30 minutes long.
After uploading, your video will go into a queue to be translated. This can take minutes or hours depending on how many people are using the service when you are. The test video we used was 57 seconds long and took just over 10 minutes to upload and render.
The underlying mechanism is a branch of computer science known as natural language processing. It allows computers to process language as spoken by humans rather than the formulaic or mathematical languages usually used by computer systems.
Other AI Video Translation Solutions
HeyGen is far from the only AI voice translator available to video creators. ElevenLabs and Descript are companies we’ve reviewed in the past for their AI Video editing and voiceover capabilities, but there are dozens of others out there as well.
Elevenlabs currently translates into 29 languages and allows you to use video links from YouTube, X, TikTok, Vimeo or a URL. While the translation itself is adequate, there’s no real attempt at lip synching and the tone of the original speaker didn’t seem to come through as well as with HeyGen.
Descript boasts 20+ languages. They market this as “dubbing” which usually means that there is no lip sync or tone preservation built in. Unfortunately, you need to pay for the Creator ($24) or Business ($50) subscription to get 30 minutes or 2 hours of dubbing respectively. The free and hobbyist level subscriptions don’t include translation.
Adobe Firefly is a suite of generative AI technologies from the makers of Premier and Illustrator. To get access to translation features, you’ll need a paid subscription. The $9.99 standard plan lets you translate 6 minutes of video while the $29.99 Pro plan gives you 23 minutes. The premium plan gives you up to 166 minutes of translation. It doesn’t work with URLs, you’ll need to have the file ready to upload.
All of these solutions work for brands that do some translation, but don’t need it all the time. Subscriptions include access to a suite of other tools that video creators or marketers will likely find useful. More purpose-built tools like Smartcat are better suited to brands that do a lot of translation, because they charge a higher price ($99 per month to start) and deliver more focused services.
Tips for Successful AI Video Translation
Whichever AI video translation tool you decide to use, the same best practices apply.
- Know your audience. Don’t just translate, localize. Localizing is the act of considering cultural nuance to ensure your script resonates in a specific area. The whole point of translating a video is to reach a broader audience. Make sure you’re communicating with the accent, dialect and verbal imagery that will make sense to them.
- Beware of figurative language. A direct translation isn’t always the best choice. Certain similes, metaphors and figurative language just won’t make sense in translation. If you know you’re going to translate your video, write your script with that in mind. Avoid language that might not make sense in translation.
- Double check the results. Ideally you’ll share your translated video with a fluent speaker before it is widely published. Even the best translation tools can make mistakes that would be obvious to someone who actually speaks the language. Avoid embarrassment by double-checking your results.
Start with a Quality Video
Of course, audio is just one part of a successful video. Before you go through the trouble of translation, you need a quality video with a strong script. That’s where IdeaRocket can help. We create commercials, explainer videos, and more for Healthcare, SaaS, Human Resources, and B2B organizations. Contact us for help creating a video worthy of a worldwide audience.
