MLSE and Amazon are bringing AR and VR to watching sports

A reporter on the Scotiabank Arena's training court tries out a virtual reality feature as part of the immersive basketball experience, launched by the MLSE initiative and AWS SpaceX.  It allows the viewer to see life-size 3D renderings of sports.  (credit: MLSE)

A reporter attempts a virtual reality feature as part of an immersive basketball experience, launched by MLSE Ditigal Labs and Amazon Web Services’ SportsX software. It allows the viewer to see life-size 3D renderings of sports. (credit: MLSE)

If you simply walked into the Scotiabank Arena’s gym on Monday night, you’d see a room full of reporters and developers, as well as representatives from Amazon Web Services (AWS) and Maple Leafs Sports and Entertainment (MLSE).

What you haven’t seen yet is a 3D rendering of the mannequins, representing players from the Toronto Raptors and Milwaukee Bucks, recreating their last NBA game. That is until you wear Augmented Reality (AR) or Virtual Reality (VR) glasses.

With a headset, you can immerse yourself directly in the action virtually. Players come to life in front of you, providing the opportunity to follow them from baseline to baseline, or simply sit on the field – whatever you feel gives you the best view of defensive meltdowns and offensive explosions.

“It is a glimpse into the future,” said Hamza Teheran, Head of Technology and Digital at MLSE.

MLSE Digital Labs and AWS have partnered to launch SportsX, a new research and development program. Its first initiative is to introduce AR and VR to change the way we can consume professional sports. This is for the fans’ experience and to help improve team performance, as players and coaches can relive moments to help their coaching.

As part of the Jan. 24 launch, MLSE and AWS demonstrated multiple variations of how they use AR and VR headsets to display the data they’ve collected. It allows headset wearers to either immerse themselves directly in the action, or even receive real-time stats as part of their view while watching a game — like Monday Night’s Toronto Maple Leafs vs. New York Islanders game.

“Frankly, this has not happened anywhere else in the world,” Tehran said.

“The important thing about this program is to build the future of the sport from Toronto to the rest of the world and to be able to use these great things within MLSE. We hope they can become examples for the rest of the sporting world.”

How does the technology work?

The Immersive Basketball Experience uses a combination of visual tracking body position data – such as the position of each knuckle and limb – 3D models and a video game development engine. The result is a life-size way to recreate the game, as players are transformed into a hologram around you.

For the NBA experience, MLSE and AWS use in-game cameras located in all arenas to collect necessary data. The same experience can be used for NHL games, while there are developments in other professional sports such as Formula 1 for biomechanical data acquisition, as we can expect this experience to come across in all sports.

For the NHL, the experience could be more realistic, as the league uses NHL Edge. To gather data, the NHL uses infrared technology embedded in the pucks and in the players’ jerseys — information that’s transmitted to — in this case — the AWS front site at the Scotiabank Arena. This technology allows every part of the movement in the game to be tracked, which lends itself to the “Extended Reality Stats Overlay” feature on an augmented reality or virtual reality headset.

While watching an NHL game, viewers can access an overlay that displays stats for any team, such as puck possession, speed, and distance traveled. Using the laser feature as part of the remote control that came with the headset, viewers can individually select a player, like Auston Matthews, to focus on for specific stats.

“You can enjoy the game while seeing real-time analytics and data in one place,” Teherany said, noting that this game can be used by hockey fans or anyone learning the game.

The reporter is dressed in the latest fashion

A reporter wears a “Mixed Reality” headset as he watches the Toronto Maple Leafs play the New York Islanders on January 23, allowing him to use the NHL Extended Reality Stats Overlay to access real-time stats about the game. (MLSE)

Future uses: squares in everyday life

Right now, the cost of a few thousand dollars, and the size of the headphones, are still a challenge to get them to a large group of people. Teherany says that when innovations like Apple’s augmented reality glasses appear, they will change the market, and we can expect that these features will be available in the size of your everyday glasses.

What’s changed dramatically in the market in recent years, says Eric Gales, country director for AWS Canada, is that before that you needed a lot of equipment to deliver an AR or VR experience. Now, thanks to all the information they have gathered on the cloud, they’re ready to try that experiment when there’s an accessible option to make it mainstream.

The “Extended Reality Stats Overlay” is an experience MLSE wants to give its fans inside arenas. This will help maximize the viewing experience, especially for those who are sitting high in the crowd. A similar stat experiment – without the headphone element – has also been displayed on a table-sized board at the Scotiabank Arena – which could become a staple inside their fancy suits.

Viewers inside the Scotiabank Arena can use the scoreboard to track players in real time, such as their movements, puck possession, speed and distance traveled.  (credit: MLSE)

Viewers inside the Scotiabank Arena can use the scoreboard to track players in real time, such as their movements, puck possession, speed and distance traveled. (credit: MLSE)

Outside the arena, there’s the issue of figuring out how to license these games for AR and VR use — since the streaming rights are owned by Rogers and Bell.

For the immersive basketball experience of professional NBA games, MLSE and AWS members want to extend their capabilities into people’s homes for everyday use. After speaking to several developers, a realistic timeline is 5-10 years, given the aforementioned cost and size issues of the headset.

For a life-size version for two players, you’ll need a larger, more accommodating space, explains Christian Magsisi, MLSE’s vice president of venue and digital technology. For example, if you wanted to be next to a life-size Kawhi Leonard during his famous 2019 game-winning shot, you would need a large space such as a gym or playground.

But with AR and VR capabilities, users will be able to place the experience on a smaller surface like a tabletop, helping them see the action unfold right in front of them.

Create a competitive advantage

One of the benefits of experiencing virtual and augmented reality is for professional sports organizations a part of MLSE. Magisi says it allows players and coaches to recreate and learn from pivotal moments from their games.

“It allows us to create a competitive advantage,” Magsisi said, noting that members of the Raptors and Leafs have already experimented with the technology.

Instead of needing 10 real players to recreate a specific game on an NBA court, you can use AR or VR to bring that moment to life virtually. Players and coaches can also watch or re-watch games using headphone technology that shows them important stats as the action unfolds, giving them a chance to see exactly what is contributing to their team’s play.

Gallis also notes the impact this can have on enhancing their understanding of injuries, and how to avoid them to enhance player safety. This kind of innovative data and research has been a focus of the NFL in partnership with AWS since 2019.

Get people involved

Besides enhancing viewing experiences and helping teams gain a competitive edge, SportsX itself is based on the concept of engaging communities. They want fans to help guide them towards future ideas through submissions, but also to experiment with AR and VR experiences.

“We’ve been watching sports the same way for a long time. Giving people virtual reality glasses is a big change. So we need to bring comfort to people. So even though it may be available soon, I think it will take some time to People really embrace.”

Those interested can start immediately by signing up at to test the technology out for themselves.

More from Yahoo Sports

Leave a Comment