Human Assist

A suite of new AI and augmented reality tools to help workers make better and more consistent decisions

Turning workers into experts through skills-transfer and automation

Reducing labour shortages as workers are upskilled into high-quality jobs

Delivering a step-change in profitability for producers and exporters

Fully automating some orchard activities

The ‘Multipurpose Human Assist’s new technologies will be exportable as it directly addresses the global need for skilled and accurate manual labour in high-value orchards and vineyards. It has the potential to be applied to other industries and crafts that are also relying on skilled and accurate labour.

What is the need?

The orchard/vineyard industries say they need:

• Rapid and accurate decision-making by workers in the field
• Improved human operator consistency
• Skills-transfer to address the shortage of experienced workers
• Automation technology to ease the upward cost pressures
• Greater predictability of the quality and yield higher value fruit.

How will it work?

We will combine four key elements:

• Data: Rich database on the crop, tree and workers over space and time. AI tools to optimise orchard/vineyard decisions and actions

• Sensor Fusion: Suite of machine vision, optical, acoustic and microwave sensors capturing information on the crop and operators

• Human Assist Tools: Data and knowledge gained from AI models to develop tools such as augmeted reality, virtual reality and dexterious tools to guide decision making and guide the actions of worker

• Communities: How the technology affects workers and communities will be investigated. A focus will be placed on Mātauranga Māori, cultural heritage, knowledge and IP in the co-design and adoption of new technologies.

These elements then become the Human Assist Platform

The project will focus on 3 case studies:

1. Apple fruitlet thinning

2. Grape pruning

3. Blueberry harvesting

The platform technology can then be adapted to other orchard and vineyard tasks. Down the track, we can link this to automated machines.

How will growers, exporters and New Zealand benefit?

By 2028, this programme is expected to deliver $416 million p.a. in economic benefits to New Zealand in:

• High-value manufacturing exports through human-assist systems and fully automated systems
• High-value fruit crops/wine export receipts through higher yields and improved quality
• Domestic high-value fruit production through reduced labour costs.

What new technology will the project deliver?

A new, rich synchronised database of vine/tree structure and operator’s movements and activities including:

•  Assessing plant development across seasons (temporal)
•  Assessing variations across the orchard/vineyard (spatial)

A new technology platform for:

• Automated learning of “best practice” actions from expert operators leading to a digital model of the expert operator
• Human-assist system (AR/VR) and wearable hardware to inform, guide and run quality control of manual orchard tasks
• Digital action models to guide robots for task automation.

From here, growers will be able to:

• Support less experienced workers with augmented reality so they make better and more consistent decisions that will enhance crop quality
• Inform autonomous robots to ultimately automate the tasks.

Looking ahead

We will fill a significant gap in global fruit industries by creating a high-quality tech augmented workforce (e.g. pruning, thinning, harvesting).

• New wearable sensors will assess the crop canopy in front of an experienced operator (e.g. branches, buds, fruitlets) and monitor their manual actions (gestures, events, tasks)
• New AI methods to learn actions from sensor data
• We can apply the automation platform to other orchard and vineyard operations
• Being highly transferrable and addressing a global need, the technology has significant export potential
• The new knowledge New Zealand gains will increase our science capabilities and put us into the R&D frontier of AI‑based utomation for horticulture
• Learnt knowledge will support inexpert staff using interface methods such as AR; reducing operator inconsistency
• We will understand technology adoption in the workforce and how automation affects hoticultural communities
• We will advance Human Assist technologies into a control system for automated robots
• We will better understand the constraints around technology adoption and consider how we can support social benefits from the introduction of new AI technologies.

Contact us

For further inquiries, please contact us maaratech@auckland.ac.nz