Blenheim
Matt Mattar has been on the UC Vision team for just over a year. We talked to him about what he does with programming robots for computer vision research which are used in vineyards. He talks about how he works on the front end of a database and doing the navigation stack for the robot. Fieldwork is a big part of the Maaratech project which he is working on. He describes what it is like out in the field.
Blenheim
Niels van Antwerpen did a 4th year project was to develop a detection and identification system. He trained an AI system to detect plants in a vineyard in order for the viticulturalist to go out into the vineyard and take photos and compare with photos in the past.
Niels has enjoyed Computer and software engineering at UC and working with the UC Vision team.
Watch the video of Niels talking about his time working with UC Vision.
Arboriculture
In late November the UC Drone project team travelled to the University of Auckland to test the DJI Matrice drone in their very impressive wind tunnel.
The DJI Matrice drone was transported from the University of Canterbury to the University of Auckland, minus the batteries. We then borrowed some Auckland based batteries and flew the drone against a wind in the tunnel. The drone has impressive stability against the wind.
Presentations were given by both the UC and UoA teams and there were lab tours showing some of the amazing drones at the University of Auckland. It was wonderful to see what Professor Karl Stol and his research team are developing.
It was agreed that this was a very useful meeting and that we needed to have more in person meetings in the future as this is where a lot of research integration would happen.
Arboriculture
The drone team had a trip on 23 August to Proseed in Amberley who kindly allowed us to test our drone onsite. Our final year project team with some industry support had made a cutting tool which comes out in front of the drone to cut pine catkins.
Blenheim
The Maaratech Team, both the University of Canterbury and University of Auckland teams, met in Blenheim last week for field work. The fieldtrip is still running this week, and it is hoped that a lot of data and insights will be gained from this trip.
The Maaratech project aims to reduce the variability of human worker performance in orchards and vineyards, using augmented reality technology to help inexpert workers make expert decisions, and reduce the risk associated with a transient workforce, using robotic automation. We will create new technologies that capture and convert data about plant structure, and the activities of expert humans, into smart decisions and actions in fruit growing e.g which grape vine to prune and how much to remove.
Arboriculture
Last week on 14 June, an Industry event was held for Professor Richard Green’s $10mil drones with tools project which is funded by MBIE. The project has a key research team from CSSE and Mechanical Engineering, University of Canterbury, let by Prof. Dan Zhou, as well as a team led by Assoc. Prof Karl Stol from the University of Auckland. Industry partners were invited to see progress on this interesting project which started last October.
The event was located at the UC Club where research presentations were given by the key researchers as well as lab tours of the Mechanical Engineering wind tunnels and the Drone lab located in Computer Science and Software Engineering department. We were lucky to have warm sunny winter weather and there were many interesting discussions had about the research.
This project will span five years and aims to enable drones to use tools with high precision in mid-air in difficult changeable settings. The goal is to create technology that can be used in forestry, testing power lines or in the construction industry
Arboriculture
Today Sam Schofield and two researchers from Victoria University, Professor Peter Smith and Ikram Singh were collecting data to test the use of stochastic differential equations to model the motion of a drone during flight.
The drone was configured to fly autonomously along a specified flight path, while a motion capture system was used to measure its precise position and orientation.
This testing is part of the Drones Project funded by MBIE.
Conan
Part of the Maaratech research team travelled to Hope, Nelson in late November 2021 to collect data from an apple orchard. The team, consisting of Oliver Batchelor, Josh McCulloch and Max Andrew, ran the robot Conan up and down the rows of espaliered apple trees at Hoddys Orchards.
The camera array and robotic arm were both utilised. The arm has cameras on the end which scan the leaves, branches and apples in a circular motion, starting underneath the branch and moving around in a curve with data recorded onto the computer.
With espaliered trees the apples, branches and leaves are literally in one plane and are able to be scanned easily by Conan the robot. The hope is that machines like Conan will eventually be developed as a commercial robot to thin fruit as well as pruning and other tasks in the orchard.
Professor Richard Green wrote a report on this fieldtrip:
Nelson field trip to Hoddys Orchard
Transported Conan, UR5 robot arm and LAL leaf blower.
Set up a new laptop and did some maintenance on calibration/stereo inference code for running Auckland’s robotic arm scanning code – and successfully collected data with stereo cameras on the UR5.
Stereo training framework:
– Data loaders for a wide range of public datasets (and our own Maaratech datasets)
– Configurable augmentations (some slightly novel ones like pasting one dataset on top of another and trapezoid warping).
– Separate benchmarking methods for feature extraction, cost volume creation and 3D filtering
(The reason for focus on benchmarking is to establish a happy medium in terms of performance and quality, aside from HSMNet (which we currently use) none of the existing works is remotely in real-time for higher resolution images.) To improve the robustness/accuracy, we have captured some multi-view datasets (botanic gardens) and working on methods on how to best capture data for easiest reconstruction.
Better quality reconstructions on a variety of scenes, looking at using the sparse point cloud for better initialisation and establishing priors for ray sampling. Potential use of NeRF variant focused on surface reconstruction to avoid noisy surfaces.
Skeleton extraction with Harry Dobbs we are focused on estimating local radius and branch direction, for the purposes of using this to disambiguate overlapping branches during skeletonization. i.e. skeletonization becomes a very simple last step with this information
Maps of vineyards and orchards
We have developed a tool for quickly creating maps of vineyards and orchards. These maps can then be exported in our standard map format for uses in the path planner on Conan and in gazebo simulations:
As an example, the robot has been instructed to scan rows 1,2,5, and 6 – where the path planner has been updated.
Another map contains two blocks (in different colours), where Conan has been instructed to scan one row from each block. After scanning the first row Conan navigates to the second block.
Check out photos of our UC Computer Vision Research team and Conan the robot in action.
