Maaratech – UCVision

Maaratech Update July 2023

In July 2023 we conducted our Maaratech field work in the Marlborough region.

Using our robot Genie, we scanned both sides of 110 plants in the Cloudy Bay vineyard.

The HDR capture coupled with the close alignment between each sides meant we got the best photos yet.

We also scan 1.5ha (approx. 3,000 vines) of vines at our secondary site:

In this vineyard we made use of the row following to achieve consistent distance from the vines.

We also tested our remote control of the vehicle. Here is a picture of Matt down the road having a coffee and monitoring the robot. The robot was autonomously scanning and no one was on site to monitor it.

Richie has now mounted all the camera’s on the adjustable camera bar onto Genie (see picture below). They still need some adjustment. Hopefully we will get to test this new set up in Lincoln in July.

Meet our Team – Matt Mattar

Matt Mattar has been on the UC Vision team for just over a year. We talked to him about what he does with programming robots for computer vision research which are used in vineyards. He talks about how he works on the front end of a database and doing the navigation stack for the robot. Fieldwork is a big part of the Maaratech project which he is working on. He describes what it is like out in the field.

Meet our Team – Niels van Antwerpen

Niels van Antwerpen did a 4th year project was to develop a detection and identification system. He trained an AI system to detect plants in a vineyard in order for the viticulturalist to go out into the vineyard and take photos and compare with photos in the past.

Niels has enjoyed Computer and software engineering at UC and working with the UC Vision team.

Watch the video of Niels talking about his time working with UC Vision.

Maaratech Fieldtrip to Blenheim 2022

The Maaratech Team, both the University of Canterbury and University of Auckland teams, met in Blenheim last week for field work. The fieldtrip is still running this week, and it is hoped that a lot of data and insights will be gained from this trip.

The Maaratech project aims to reduce the variability of human worker performance in orchards and vineyards, using augmented reality technology to help inexpert workers make expert decisions, and reduce the risk associated with a transient workforce, using robotic automation. We will create new technologies that capture and convert data about plant structure, and the activities of expert humans, into smart decisions and actions in fruit growing e.g which grape vine to prune and how much to remove.

Maaratech Fieldtrip to Nelson

Part of the Maaratech research team travelled to Hope, Nelson in late November 2021 to collect data from an apple orchard. The team, consisting of Oliver Batchelor, Josh McCulloch and Max Andrew, ran the robot Conan up and down the rows of espaliered apple trees at Hoddys Orchards.

The camera array and robotic arm were both utilised. The arm has cameras on the end which scan the leaves, branches and apples in a circular motion, starting underneath the branch and moving around in a curve with data recorded onto the computer.

With espaliered trees the apples, branches and leaves are literally in one plane and are able to be scanned easily by Conan the robot. The hope is that machines like Conan will eventually be developed as a commercial robot to thin fruit as well as pruning and other tasks in the orchard.

Professor Richard Green wrote a report on this fieldtrip:

Nelson field trip to Hoddys Orchard

Transported Conan, UR5 robot arm and LAL leaf blower.

Set up a new laptop and did some maintenance on calibration/stereo inference code for running Auckland’s robotic arm scanning code – and successfully collected data with stereo cameras on the UR5.

Stereo training framework:

– Data loaders for a wide range of public datasets (and our own Maaratech datasets)

– Configurable augmentations (some slightly novel ones like pasting one dataset on top of another and trapezoid warping).

– Separate benchmarking methods for feature extraction, cost volume creation and 3D filtering

(The reason for focus on benchmarking is to establish a happy medium in terms of performance and quality, aside from HSMNet (which we currently use) none of the existing works is remotely in real-time for higher resolution images.) To improve the robustness/accuracy, we have captured some multi-view datasets (botanic gardens) and working on methods on how to best capture data for easiest reconstruction.

Better quality reconstructions on a variety of scenes, looking at using the sparse point cloud for better initialisation and establishing priors for ray sampling. Potential use of NeRF variant focused on surface reconstruction to avoid noisy surfaces.

Skeleton extraction with Harry Dobbs we are focused on estimating local radius and branch direction, for the purposes of using this to disambiguate overlapping branches during skeletonization. i.e. skeletonization becomes a very simple last step with this information

Maps of vineyards and orchards

We have developed a tool for quickly creating maps of vineyards and orchards. These maps can then be exported in our standard map format for uses in the path planner on Conan and in gazebo simulations:

As an example, the robot has been instructed to scan rows 1,2,5, and 6 – where the path planner has been updated.

Another map contains two blocks (in different colours), where Conan has been instructed to scan one row from each block. After scanning the first row Conan navigates to the second block.

Check out photos of our UC Computer Vision Research team and Conan the robot in action.