Human Assist

Wearable devices and Artificial Intelligence systems for

orchardists and vintners to maximize fruit value

 Collaboration between University of Auckland (lead), University of Waikato, University of Canterbury,

University of Otago, Lincoln Agritech, and Plant and Food Research

A suite of new AI and augmented reality tools to help workers make better and more consistent decisions
Turning workers into experts through skills-transfer and automation
Reducing labour shortages as workers are upskilled into high-quality jobs
Delivering a step-change in profitability for producers and exporters
Fully automating some orchard activities

The ‘Multipurpose Human Assist’s new technologies will be exportable as it directly addresses the global need for skilled and accurate manual labour in high-value orchards and vineyards. It has the potential to be applied to other industries and crafts that are also relying on skilled and accurate labour.

What is the need?

The orchard/vineyard industries say they need:

Rapid and accurate decision-making by workers in the field
Improved human operator consistency
Skills-transfer to address the shortage of experienced workers
Automation technology to ease the upward cost pressures
Greater predictability of the quality and yield higher value fruit.

How will it work?

We will combine four key elements:

Data: Rich database on the crop, tree and workers over space and time. AI tools to optimise orchard/vineyard decisions and actions

Sensor Fusion: Suite of machine vision, optical, acoustic and microwave sensors capturing information on the crop and operators

Human Assist Tools: Data and knowledge gained from AI models to develop tools such as augmeted reality, virtual reality and dexterious tools to guide decision making and guide the actions of worker

Communities: How the technology affects workers and communities will be investigated. A focus will be placed on Mātauranga Māori, cultural heritage, knowledge and IP in the co-design and adoption of new technologies.

These elements then become the Human Assist Platform

The project will focus on 3 case studies:

1. Apple fruitlet thinning

2. Grape pruning

3. Blueberry harvesting

The platform technology can then be adapted to other orchard and vineyard tasks. Down the track, we can link this to automated machines.

How will growers, exporters and New Zealand benefit?

By 2028, this programme is expected to deliver $416 million p.a. in economic benefits to New Zealand in:

High-value manufacturing exports through human-assist systems and fully automated systems
High-value fruit crops/wine export receipts through higher yields and improved quality
Domestic high-value fruit production through reduced labour costs.

What new technology will the project deliver?

A new, rich synchronised database of vine/tree structure and operator’s movements and activities including:

  •  Assessing plant development across seasons (temporal)
  •  Assessing variations across the orchard/vineyard (spatial)

A new technology platform for:

  • Automated learning of “best practice” actions from expert operators leading to a digital model of the expert operator
  • Human-assist system (AR/VR) and wearable hardware to inform, guide and run quality control of manual orchard tasks
  • Digital action models to guide robots for task automation.

From here, growers will be able to:

  • Support less experienced workers with augmented reality so they make better and more consistent decisions that will enhance crop quality
  • Inform autonomous robots to ultimately automate the tasks.

Looking ahead

We will fill a significant gap in global fruit industries by creating a high-quality tech augmented workforce (e.g. pruning, thinning, harvesting).

New wearable sensors will assess the crop canopy in front of an experienced operator (e.g. branches, buds, fruitlets) and monitor their manual actions (gestures, events, tasks)
New AI methods to learn actions from sensor data
We can apply the automation platform to other orchard and vineyard operations
Being highly transferrable and addressing a global need, the technology has significant export potential
The new knowledge New Zealand gains will increase our science capabilities and put us into the R&D frontier of AI‑based utomation for horticulture
Learnt knowledge will support inexpert staff using interface methods such as AR; reducing operator inconsistency
We will understand technology adoption in the workforce and how automation affects hoticultural communities
We will advance Human Assist technologies into a control system for automated robots
We will better understand the constraints around technology adoption and consider how we can support social benefits from the introduction of new AI technologies.

Contact us

For further inquiries, please contact us