Photonics Charts Its Course in Helping Athletes Reach Peak Performance

MICHAEL EISENSTEIN, CONTRIBUTING EDITOR

Michael Matter’s journey into sports technology began with a golf ball. More specifically, a photo of a golf ball that was captured at its precise moment of impact with a club head. The photo — shot by Harold “Doc” Edgerton, the legendary high-speed photography pioneer — showed in detail how the connection flattened the round, rigid surface of the ball.

As a boy, Matter was an enthusiast of both photography and electronics and aimed to replicate Edgerton’s stroboscopically achieved imaging feats. “I figured out how to modify a $20 electronic flash to get it from being about a 2-ms flash duration down to about 10 μs,” Matter said. In subsequent years, Matter worked with Edgerton at MIT. Eventually, he parlayed his engineering expertise into the Edgertronic series of high-speed cameras.

Courtesy of iStock.com/bmcent¹.

Decades later, these cameras — produced by Sanstreak, the company that Matter founded in 2012 — are a mainstay of the baseball world. This imaging technology is a leading example of how sophisticated optics and photonics technologies are literal game-changing forces in sports. Fittingly, the post-2015 period in Major League Baseball (MLB) is sometimes referred to as the “Statcast era”; high velocity and precision control are making pitchers increasingly difficult to hit, and, as a result, hitters are inviting nontraditional, technology-based tools to improve their performance. The Statcast name is also an acknowledgment of how this now- ubiquitous platform for camera-based tracking and analysis of gameplay has transformed the game.

Beyond the baseball diamond, athletes in many other sports are enjoying a similar high-tech boost. Basketball, soccer, American football, and gymnastics are among other sports using motion capture, lidar tracking, and VR to optimize performance and develop winning strategies, and team managers, owners, and coaches are scrambling to stay at the leading edge of the technology curve. “There’s a huge number of data scientists each year being recruited by [soccer] clubs,” said Ian Cowling, CTO of Bicester, England-based Sportlight Technology. “But there are also a lot of problems to solve, and I’ve not heard of a club who’s got enough data scientists yet,” Cowling said.

Caught in the act

Perhaps unsurprisingly, Matter’s original vision for Edgertronic in the sporting world was for the technology to aid golfers in refining their swing. But the super-slow-motion analysis enabled by his cameras, which can deliver 720p-resolution images at >700 fps, is overkill here. “Ninety-nine percent of golf is like, ‘Is your swing good? Keep your head down. Did you follow through?’” Matter said. Indeed, several companies, such as motion capture and 3D visualization technology developer Uplift, now offer software that enables golfers to carefully analyze their swing based on videos captured with a standard iPhone or iPad.

Edgertronic burst onto the MLB scene in 2015 when a young pitcher started experimenting with Edgertronic, using the ultrahigh-speed images to study his posture, grip, and release. “He said this camera let him invent new pitches,” Matter said. Enthusiasm for the technology quickly spread, much to the benefit of the company. “You can pay half a million dollars to a New York ad agency and they wouldn’t come up with anything as good,” Matter said.

KinaTrax’s markerless motion capture technology for baseball captures ultrahigh-resolution video at high frame rates to deliver insights on mechanics, movements, and other parameters to pitchers and hitters. Courtesy of

By 2019, the technology had become widespread in the Big Leagues, and MLB published an article calling the cameras one of the technologies that is “changing baseball.” According to Matter, most teams have three or four cameras, though he said that one organization uses more than 200, with installations at its training and minor league facilities. The cameras are routinely used in-game to allow players and coaches to carefully review the posture and performance of both pitchers and hitters at every at-bat.

This technology can also be combined with other technologies used in the baseball world, such as the radar-based pitch tracking systems developed by Rapsodo, which MLB uses to measure ball speed and trajectory in-game. This pairing reveals insights into how subtle changes in timing, posture, grip, and/or movement influence the motion of a thrown or hit ball.

Player tracking up to speed

One limitation of Edgertronic technology is that it can only monitor individual players in a relatively fixed position. Though Matter said that the cameras have found situational use in other sports — for example, foul shot and three-point shooting in basketball and placekicking in football — other systems are needed to capture the hectic multiplayer action that is characteristic of most team and individual sports.

Many top leagues currently use large arrays of networked cameras strategically positioned at key sites in stadiums or arenas. These cameras capture complete game action, in high detail, and the resulting data is interpreted algorithmically to identify specific players and reconstruct their movements. Major companies in the “player tracking” space include Tracab, whose systems are widely used in soccer, and Hawk-Eye Innovations, which is currently employed by the MLB Statcast system and is also a strategic partner of the National Basketball Association (NBA).

VR-based training programs from Rezzil simulate competition scenarios. The technology can artificially augment the capabilities of a user’s opponent to improve situational performance. Courtesy of Rezzil.

Output data from KinaTrax’s “kinematic sequence” charts the relationship between frame and angular velocity as it relates to the motion mechanics of a baseball player in action. Courtesy of KinaTrax.

Yet while cameras thoroughly document the action of a game, they cannot always precisely quantify the speed and trajectory with which players move and change direction. Cowling and colleagues at Sportlight see lidar as a powerful adjunct for this purpose. The detection and ranging technology delivers accurate measurements of object position and velocity by analyzing the reflection of invisible infrared lasers.

Lidar is widely known and used in the automotive sector, and the rapid growth of lidar in this industry has been a boon to Sportlight’s penetration into athletics, resulting in falling costs for sensors that also deliver steadily improving spatial and temporal resolution. Its current lidar units collect >1 million measurements per second, enabling the simultaneous measurement of every player’s movement on the field. One peer-reviewed study from 2022 confirmed the precision of its system: The level of error on player velocity readings was on the order of ~10 cm/s, according to the study¹.

Lidar is not the only technology used to provide such analysis. Certain teams use GPS for this task, Cowling said. However, GPS performance can be inconsistent in different areas of the field, leading to incomplete data on player movement. “Sometimes the wide players aren’t tracked for 60% to 70% of the game because GPS signals don’t get through,” Cowling said. In contrast, one or two lidar units combined with a standard camera system can capture and quantify every movement.

“‘What condition are my athletes in, how have they moved over the week, how are they moving at mid-season compared to the start of the season?’ Those are the sorts of questions that we can work on,” Cowling said. According to Cowling, 19 of the 20 English Premier League soccer teams currently use Sportlight, and the company recently began working with the NBA to assess the technology’s applicability to basketball.

Picking up the pieces

To obtain the most accurate performance analysis of an elite athlete, one must carefully analyze each component piece of the athlete’s body as it moves through space and rotates while pitching, catching, kicking, or leaping. Motion capture technologies, which algorithmically analyze video footage and reconstruct a virtual “skeleton” that mimics the anatomy of an athlete in action, have immensely benefited this sort of biomechanical analysis. The resulting insights can help players hone their performance and track progress in recovering from injury and reduce the risk of future traumas.

Information displayed on Fujitsu’s Judging Support System (JSS) screen (top). The left side of the image shows the video data and the height and body angle of the elements performed by the athlete in a timeline. The right side displays the athlete’s movements in 360°. The lower part of the image displays each element performed and a characterization of its difficulty.

Fujitsu’s Judging Support System (JSS) tracks the movement of a gymnast in action (middle) and precisely captures the position of individual body parts in relation to the uneven bars (bottom). Courtesy of Fujitsu.

Many biomechanics labs and training facilities use marker-based systems that require athletes to undergo precise tagging with labels, which the analytical software can use to recognize and track individual body parts for subsequent reconstruction. Marker-based methods are well established and can be powerful tools, but they also require expert supervision and may not capture real-world performance. These require athletes to undergo precise tagging with labels that the analytical software can use to recognize and track individual body parts for subsequent reconstruction. “It’s unnatural,” said Scott Coleman, director of sales and marketing at KinaTrax. “They’re in their underwear, they have all this stuff on, and then it takes hours and hours.”

KinaTrax is one of several companies offering markerless motion capture, with a primary focus on baseball. According to Coleman, approaches such as those developed by KinaTrax have been enabled by the development of sophisticated digital cameras that capture ultrahigh-resolution video at high frame rates; KinaTrax’s cur- rent systems operate at 600 fps. The systems analyze the acquired data via an AI algorithm that has been trained to recognize key landmarks on the players’ bodies and convert them into a 19-segment biomechanical model. These systems can analyze player performance in actual games without disrupting the athletes’ routines.

A typical KinaTrax system setup for baseball involves positioning eight to 16 cameras between the left field and right field foul poles and home plate. The technology generates complete biomechanical profiles for every pitch and at-bat in a game. Among the measurables that the system gauges, Coleman said, are player kinematics and kinetics, temporal and spatial values, step width, step length, and stride.

KinaTrax is currently used by 20 MLB teams, along with numerous minor league teams and training facilities, Coleman said. The company aims to extend motion capture to every single baseball player on the field during the next several years. It is also looking into extending its offerings into other sports, with preliminary work underway in the NBA and National Hockey League. These sports will pose a thornier challenge than baseball, however, with more movement and opportunities for occlusion of the camera’s view by other players, officials, and fans.

AI assists gymnastics judging

Athletes and coaches are not the only ones benefiting from markerless motion capture. Japanese technology company Fujitsu is demonstrating how this technology can turn the art of judging gymnastics into more of a precise science.

According to Shoichi Masui, a senior research expert at Fujitsu, the company first launched a project to optimize athletics judging following a remark made by the head of the International Gymnastics Federation (FIG) during a meeting in 2015 with Hidenori Fujiwara, head of Fujitsu’s sports business division. “He jokingly said, ‘In the future, maybe robots will be doing the scoring,’” Masui said. “[But] Fujiwara took the joke seriously and built a prototype.”

Fujitsu debuted its first-generation Judging Support System (JSS) a year after the meeting. The system used lidar to scan gymnasts’ bodies and generate a 3D contour map, which was then computationally converted to a simplified skeleton that can be analyzed to assess posture and form. During the next few years, the then-high costs of lidar drove the company to transition to a multicamera-based system, coupled with a more sophisticated AI-based algorithmic platform.

According to Masui, Fujitsu also grappled with significant imaging challenges that confounded analysis in earlier iterations of JSSs. These included the blurring of joint positions as estimated by the AI-assisted skeleton recognition algorithm, and the preparation of the training data for the AI system. The Fujitsu team ultimately resolved these bottlenecks with algorithmic tools for error-correction and the production of computationally generated synthetic — though highly photorealistic — training data.

Fujitsu first deployed JSS for pommel horse competitions. The concept was that this event would provide a viable testbed because of the relatively limited performance area compared to other gymnastics events. “We later realized that we had actually started the development with the most difficult apparatus even for gymnastics judges,” Masui said.

But the system proved to be a success nonetheless, and Fujitsu has been a partner with the FIG since the start of its development in 2016. In 2019, Fujitsu provided its first official showcase of the JSS at the Artistic Gymnastics World Championship in Stuttgart, Germany. At the time, the platform was limited to only four events — pommel horse, still rings, and men’s and women’s vault. Fujitsu has subsequently expanded JSS to enable judging support for all 10 standard gymnastics apparatuses, and Masui said that the FIG has expressed its intent to continue to use this platform to assist its human judges in future events.

Lidar and AI developer Sportlight Technology uses a camera system paired with multiple lidar units to capture and quantify every in-game move from a soccer team. Courtesy of Sportlight Technology.

The implementation of the technology, Masui said, shows a combination of technological and practical innovation. For judges at major competitions, he said, “long hours spent deliberating are the norm, but a technology that offers objective information to help support scoring could help streamline that process.”

Getting into the game

Finding better and smarter ways to train is a universal goal for athletes and coaches. The challenge is to find regimens that realistically test and bolster the skills that players need on game day.

“Players don’t generally feel the pressure when they’re training,” said Andy Etches, cofounder and sports director of Manchester, England-based Rezzil. The company is developing VR-based training programs that simulate real competitions, or even crank up the difficulty by artificially augmenting the performance of a user’s opponents. “If they’re playing in a friendly game, they know they’re playing in a friendly game, and so getting them to react in a natural way is really difficult,” Etches said.

It takes a lot of refinement to make VR training appealing for athletes. Etches said that some simulators focus purely on what the players see but do not incorporate their physical movements. This level of artificiality takes them out of the experience and can also contribute to unwanted side effects such as motion sickness, which could deter many new users from embracing the simulator.

“We use devices that are capable of six degrees of freedom and that have high refresh rates and enough computing power behind them to give realistic responses,” Etches said. Since the VR setup is designed to respond to actual player movement, it can be combined with biomechanical analysis to provide additional layers of insight into player performance.

The company’s current focus is on soccer, and Etches said that every team in the English Premier League is currently using Rezzil as a part of their training regimen. The company has even designed drills tailored to build essential skills such as scanning — the rapid visual information-gathering process that players perform before making a play. VR also offers a safer way to practice skills that are essential in-game but potentially harmful if practiced repeatedly off-field, such as heading a ball. Rezzil has also paired with the NBA and at least one football team in the U.S. to build specialized regimens ftor basketball and football, respectively.

Etches is excited about how the platform might bring fans closer to the game, too. The company recently signed a deal that will allow them to pull the data from future matches into their VR platform, allowing both professionals and amateurs to relive those games firsthand.

“A lot of fans sit at home and go, ‘Oh, I could have scored that,’ and we’ll go, ‘OK then, here’s the ball coming at you at 80 mph,’” Etches said. “I think it will bring people closer to the game.”

References

1. T. Bampouras and N.M. Thomas (May 2022). Validation of a LiDAR-based player tracking system during football-specific tasks. Sports Engineering, Vol. 25, No. 8.