Aotearoa’s apple industry struggles to maintain the skilled workforce required for fruitlet thinning each year. Skilled labourers play a pivotal role in managing crop loads by precisely thinning fruitlets to a desired number to achieve the desired spacing for high-quality apple growth. This is a complex task that requires accurate mapping of the fruitlets along each branch. This paper presents a novel vision system capable of mapping the orientation and clustering information of apple fruitlets as a human expert does. The vision system has been validated against data collected from a real-world commercial apple orchard. The results show an improved counting accuracy of 83.97% on prior implementations, an orientation estimate accuracy of 88.1%, and a clustering accuracy of 94.3%. Future work will utilise this information to determine which fruitlets to remove and then robotically thin them from the canopy.