CREATION
December 2018
Using NewTek TriCaster allows live editing between animation and live action
angles and ISO-recording of all feeds as well as outputting the programme feed to
a thumb drive for sharing.
Bringing a Turtle to life requires Adobe Character Animator streaming into the
Unreal Engine via NDI in real time.
tion, Nickelodeon launched its
Nickelodeon Entertainment Lab,
which will spearhead research and
development efforts around new
technologies for the network and
its audience. At this year’s Comic-
Con Convention in San Diego,
Nickelodeon Entertainment Lab
brought the new world of Rise of
the Teenage Mutant Ninja Turtles
to life.
Chris Young, senior vice-
president of the Nickelodeon
Entertainment Lab, headed up
this VR project, which is designed
to create a PR splash to publicise
the reboot of the Teenage Mutant
Ninja Turtles franchise. By creating
a virtual press junket, Young was
hoping to be able to immerse jour-
nalists into the Turtles’ new world.
“We wanted journalists and
superfans at Comic-Con to have
this opportunity to step inside the
Turtles’ art-directed world,” Young
says. “We wanted them to get a
first-hand look at the new show,
and be able to interview Mikey and
Donnie in VR. Our plan was to
film the interview using live-action
cameras composited with gaming
footage in mixed reality.”
The project is completed using
Adobe Character Animator, Epic
Games’ Unreal Engine — which
supports NDI, NewTek’s IP video
protocol — and NewTek TriCaster.
Young elaborates: “It all starts with
the Adobe Character Animator.
We stream that into Unreal Engine
using NDI technology to get it into
the game. So, the person wearing
the VR headset in the game is see-
ing the animated Turtles streaming
over NDI in the game. From there,
we’re also streaming NDI into live
compositing software, where we’re
compositing the footage together
— both virtual camera shots of the
Turtles, and live action footage of
the journalists.
“All of those signals were then
live-streamed back over NDI to
the NewTek TriCaster system.
Using TriCaster, we were able to
live edit between all the animated
and live-action camera angles, as
well as record ISO-feeds of all the
different angles, in addition to the
programme edit. Then, once the
interview was over, we were able
to throw the programme feed
on a thumb drive to give to the
journalists.”
Besides creating new viewing
experiences, these graphic systems
and virtual sets have already been
used in TV broadcasting for ap-
plications like news and elections
coverage. For instance, during
elections coverage, presenters can
have the election data displayed
along with them, in context and in
real time, simplifying the complex
election data within graphics for
audiences’ ease of understanding.
Miguel Churruca, marketing and communications director of Brainstorm: “The
integration of live graphics into real images or virtual sets is both a challenge and
an opportunity; a challenge because designers need to think on how to integrate
live images along with their creations, and an opportunity because the result is
normally far more compelling to the audience.”
Miguel Churruca, marketing
and communications director of
Brainstorm, comments: “AR, in its
vast variety of flavours, is a great
tool for enhancing the informa-
tion displayed in a broadcast pro-
gramme while adding spectacular
content. Raw data can be displayed
more attractively when placed in
context with the presenter or by
adding interactivity. Also, today’s
availability of enormous amount
of data requires visually attractive
ways to present these data to the
audience at home.
“In this sense, data-driven
graphics can enhance TV shows
when added to real shots in context
as AR content, and also the latest
3D and photorealistic virtual stu-
dio technologies can provide even
better results for engaging the au-
dience. Therefore, the audience at
home can see how presenters can
interact with complex graphics or
information, making storytelling a
lot more visually attractive.”
While this technology is in-
creasingly being applied in news,
sports and entertainment pro-
grammes, he points out the key
in creating “good” AR content en-
compasses the integration between
the real and virtual objects with
the environment. Photorealism is
“one of the toughest challenges”,
he identifies, because what really
makes the difference for the view-
ers is to be unable to tell whether
the images they are watching are
real videos or digital renders.
To make graphics “an integral
part” of the broadcast content,
some key considerations when de-
signing and creating these virtual
graphics require the close collabo-
ration between graphic designers
and other departments, such as
virtual studio operators.
This is largely due to the need
for additional hardware and soft-
ware solutions, because during
live shows where it involves virtual
sets, it often requires complex in-
stallations such as tracked cameras
and cranes, Churruca says. “From
a designer’s perspective, the inte-
gration of live graphics into real
images or virtual sets is both a
challenge and an opportunity; a
challenge because designers need
to think on how to integrate live
images along with their creations,
and an opportunity because the
result is normally far more com-
pelling to the audience.”
Brainstorm has developed
Aston, a motion graphics creation,
CG and playout system. Aston is
designed for robust on-air ope-
rations, allowing designers to
create, manipulate, animate and
perform changes even when on-
air. Using Aston with InfinitySet,
Brainstorm’s virtual set and AR so-
lution, users can create “complex
and realistic” AR environments,
joining together live characters,
virtual or real environments,
data-driven graphics, and other
design elements to create the live
experience for viewers.
For Vizrt, its AR graphics solu-
tions have been used by broadcast
and media companies worldwide
Thairath TV,
a local Thai
TV channel,
utilised Vizrt’s
AR graphics
solutions to
create AR
and virtual
graphics for
the coverage
of a cave
rescue mission
in Thailand in
June this year.
11
for sports and special events cov-
erage. In the recent case of the
two-week long cave rescue mission
in Thailand, Thairath TV, a local
Thai TV channel, utilised Vizrt’s
solutions to create AR and virtual
graphics surrounding the story.
With almost no access to live foot-
age from the scene, audiences were
able to understand and share the
experiences of the stranded boys
as well as efforts by the rescue
teams using virtual graphics, AR
and MR.
“There was an almost com-
plete ban on live coverage from
the ground with very little visu-
als of the cave, and the mission
was available to the audience,
creating great anxiety and inter-
est surrounding the story,” Vizrt’s
Krishnan explains. “Virtual graph-
ics and AR were used almost by all
channels who were actively cover-
ing this story, and having the story
put together from various reports.
“Some very impressive render-
ings of the graphics with presenter
inter actions helped to bring con-
text to the severity and danger of
the mission, simulate the rescue
attempts, while some others help
to visualise and present the inter-
nal structure and challenges of the
cave itself.”
Over the course of the cover-
age of the rescue mission, Thairath
TV received “great reviews and
improved viewership”, according
to Krishnan, with the target rating
point (TRP) of their news pro-
gramming surging up to the top
with an improvement of over eight
points from this story production.
“Virtual graphics is a powerful
medium to help with storytelling,”
he concludes. “Good storytelling
is fundamental to any broadcast
production, and it all starts with
good planning. Visual graph-
ics, animations, and any other
form of graphics or visualisation
have to serve a basic purpose
of enriching the storyline while
helping to enhance the ability to
convert and communicate with the
audience.”