Vision-Guided Robots Pick Ripe Strawberries in Seconds

HOLLY O'DELL, CONTRIBUTING EDITOR

Despite the fact that America’s consumption of strawberries is rising, fewer agricultural workers are available to pick them, creating a growing labor shortage in the strawberry fields. Picking strawberries is hard, hot work, and few want to do it. This combination of fewer workers and undesirable work is driving up the cost of these popular berries, of which Americans consume a record 3.4 lbs per person every year.

To bridge the gap between labor shortages and higher demand, Harvest CROO Robotics (HCR) of Tampa, Fla., has developed a GPS-guided robotic harvester designed to use vision-guided robots to carefully and accurately pick strawberries.

A complete setup of the Harvest CROO Robotics harvester shows the robot with cameras and gripper wheel in place. Courtesy of Harvest CROO Robotics.

The prototype HCR harvester, known as the Berry 4 (B4), can identify more than 90% of visible available berries. The system takes 8 seconds to pick a single strawberry plant and 1.5 seconds to move to the next plant. Over a 24-hour period, the harvester does the work that 30 laborers cover in an eight-hour day.

Image processing software detects berries. Courtesy of Harvest CROO Robotics.

The harvester supports up to 16 picking robots. Before the B4 gets started, a machine lays down black plastic on raised mounds in the field. This promotes water runoff so strawberries don’t rot. Once the plastic is down, B4 punches 16 consecutive holes — one from each robot — for placing the seedlings, until the harvester completes one row and pivots to the next. While the robot punches the plastic-covered ground, the harvester’s modular toolholders use GPS to accurately denote the precise location of each punch so the robot knows exactly where to return when it’s time to harvest the strawberries. Strawberry planting still relies on human laborers, but as part of the prototyping process, HCR is assessing ways its harvester can complete this task.

A gripper is placed in a rotation wheel, which uses conservation of motion concepts to ensure the strawberry is picked quickly and without damage. Courtesy of Harvest CROO Robotics.

When harvesting, the robots use a mechanism that goes down into the plant, grabbing all the branches with leaves and leaving the strawberries lying on the bed. The robots repeat this process dozens of times throughout the season because the same strawberry plant continues to grow. The system then relies on two custom Omnivision OV4682 cameras whose RGB channels distinguish between a ripe and an unripe strawberry. The software designates berry targets, using color and shape filters specially written to take advantage of the hardware. An IR channel is also deployed to see what’s happening under the skin, such as bruising.

“Strawberries are convenient for image analysis because red indicates ripeness and white indicates less ripeness,” said Rex Lee, CEO of Pyramid Imaging Inc., which has provided on- going technical advice and components to HCR since the robotics company’s inception in 2013. “The system calculates how much is red and how much is white, and it needs to establish a certain threshold of red to discern when [a strawberry] is ready to be picked.”

Each robot contains two cameras with stereo vision — which allow the robots to perceive depth — as well as a GPS to track the location of the berries, and lasers to avoid collisions. Courtesy of Harvest CROO Robotics.

The vision system currently determines that ratio, which can change from strawberry to strawberry. Once the cameras signal that the strawberries are ready for harvest, a soft rubber picker gently plucks berries in a few seconds and places them into a clamshell that goes straight to the grocery store. Strawberries that aren’t quite as ripe will be routed to secondary locations such as juicing operations.

One major problem faced by the harvester is that strawberries are constantly ripening while they are on the plant. For example, if a strawberry is left behind that is only 5% white, it could be overly ripe by the time the harvester returns for the next round of picking.

Another concern is the robots’ lack of delicacy, which can lead to bruised berries. Furthermore, dropping healthy fruit from the claws causes waste. “We use concepts of conservation of motion to minimize forces being exerted on the fruit and ensure that it’s picked rapidly and without damage,” says Bob Pitzer, co-founder and chief technology officer of Harvest CROO.

These concepts include patented mechatronics that are employed to quickly lift fruit off the black plastic and perform steps such as stem separation, initial elevation, and actuator availability for the next berry — all within a few hundred milliseconds — to enable clearing of a plant in the most efficient way. Other considerations include food safety and cost-effective design.

The cameras also track the claws for mechanical manufacturing differences. Custom real-time software interacts with targets mounted on the berry-picking claws, compensating for mechanical stack-up issues that occur in the manufacturing of machined parts.

In addition to the vision hardware and software, the harvester uses several industrial sensors for monitoring the system, including pneumatic, electrical, and engine systems over CAN (controller area network) bus. A GigE backbone incorporates navigation sensors, such as the DGPS (differential GPS) and lidar used for collision avoidance and navigation alignment. A modular design means that the harvester can be adapted to other crops.

“The HCR Harvester is being developed to supplement the ongoing labor shortage happening with specialty fruit farming, especially in the labor-intensive strawberry market,” Lee said. “We anticipate that the technologies being developed will ensure food security for the country in the future.”