Acknowledgements

We would like to sincerely thank our project supervisors, Mr Nicholas and Graham, for their guidance, encouragement, and confidence in our team throughout this module. Their mentorship has been a key factor in our success, and their trust in allowing us the freedom to explore and make decisions independently has greatly enriched our learning experiences. Our supervisor’s willingness to share their knowledge and provide constant support has been invaluable, and their dedication to our growth has made a lasting impact on our project journey.

Our gratitude also goes to the workshop technicians, Mr. Hamilton and Mr Dickson for their technical expertise and continuous assistance in ensuring our design was optimized for effective manufacturing.

We would also like to thank Ms. Annie for her efforts in managing procurement and ensuring that essential materials were acquired smoothly and on time alongside Yee Teng and Patrick for their cooperation and support in facilitating our project within the Innovation and Design Hub and providing us with valuable insights into the Hub's operations.

Special thanks to Professor Goh Cher Hiang for reviewing our project and offering constructive feedback that helped us improve further.

Finally, we extend our appreciation to EDIC for their generous funding support, which has been instrumental in bringing our project to fruition.

1 Introduction

This project is on developing a mobile manipulator for NUS’s Innovation & Design Hub to relieve staff workload. It integrates mechanical, electrical, and software design to create a unified robot system with a user-friendly GUI control and seamless system integration. Note that this report is on the overall development, integration and validation of the mobile manipulator system, with more focus on the technical development. The interim report, which has more details on the problem analysis, value proposition and initial analysis can be found in the interim report. Link to Interim Report

1.1 Context

Robots are increasingly used to relieve labour shortages and handle repetitive tasks across sectors like warehousing, healthcare, and hospitality. This is accelerating a shift from factory-only deployment to human-centric environments, with robots taking on point-to-point delivery and routine workflows as seen in figure 1.

The team had an interest in knowing where the National University of Singapore can benefit from such a system. To explore this, the team decided to choose the Innovation and Design Hub as the testing ground.

1.2 Location and stakeholders

The Hub was chosen as a location of interest as it is one of the largest workspaces within NUS with high student traffic from various disciplines to use the facilities to do discussions, fabrication or tinker. The floor plan of the space is shown in figure 2.

The project involves the Hub Technical Staff seen in figure 3, How Yee Teng (primary) and Patrick Tan Shoong Chin (secondary), as well as the Hub Technical Assistants. To gather insights, data collection through surveys and observations helped the team to understand their roles and contributions as well as understanding what problems the staff faced on the ground.

1.3 Problem Analysis and Problem Statement

To ascertain the problems further, we drafted up a problem identification table utilising the 5W1H template in Figure 4.

From our previous presentation, the problem statement defined by the team was stated that several redundant tasks are slowing down the work of the Hub staff and reducing efficiency in the Hub during the school semester. There is a need to streamline or automate the tasks.

Our design statement then for the project:

To build a system that is mobile, to travel around the Hub’s environment, can manipulate and interact with the Hub’s objects and is autonomous, the staff do not need to act as a middleman.

1.4 Value Proposition

To assist in handling tasks that cannot be handled by software changes due to the physical nature of interaction, a physical solution can be created that will be able to aid the staff in handling work as proposed in the Value Proposition Canvas in Figure 5.

The team will develop an autonomous mobile manipulator that combines navigation and manipulation: it can move through the Hub, pick and handle objects, and deliver them to staff—without requiring constant monitoring or extra devices.

1.5 Subsystems to develop

1.5.1 Design Specifications

Based on the derived problem statements and key problems identified by the staff, the design specifications were prepared. Given our limited manpower, budget of $1600 SGD and time for development of less than a year, we decided to scope down to 2 distinct tasks that the system must achieve that assists with the staff.

For task 1, the robot is to grab tools from the pegboard shown in Figure 6, put it into an onboard basket and deliver it to a table. End-effector system with tool changer capabilities enables picking tools of different geometries.

For task 2, for the staff to control and monitor signs onboard for the system to function, a user interface is to create that can link commands from the user to the relevant commands for the hardware to perform the task. A sequence diagram for the user interface is shown in Figure 7.

1.5.2 Subsystem Identification

Based on the taskings that the robot must accomplish, we have identified the key subsystems that need to be developed to bring the system to a minimum viable product, shown in figures 8 and 9.

1.6 Key changes from interim

Previously, a task 2, point – to – point delivery was to be implemented but due to time, cost and manpower constraints on the development, this was shelved as a potential future works addition to the system.

The system will be able to deliver items around the hub but will not be able to return then back, the mechanical complexity is to be scaled back to allow for more development time towards the software and main hardware deliverables.

To complement the pick and place operations such that there is consistent feedback to the system, the team opted to implement computer vision to the system with depth data, to guide the system to pick up the tools with high precision and repeatability.

1.7 Work Allocation

The updated work allocation amongst the team members is shown in the table below.

3: End effector & tool changer systems

3.1 Interim developments

To design the End Effectors for the system, design constraints must be established in terms of the actuation mode of choice, overall maximum size and the maximum mass before further analysis on the approach can be done.

From the interim presentation, the actuation mode of the end effector and tool changer were confirmed after analysing the constaints and ranking the different actuation modes based on the concept screening technique as shown in the table below. Electric actuation was selected.

From this and knowing the dimensional constraints of the mounting, the design specifications for the end effector and tool changer system were drafted and are shown in the table below.

Due to the wide variety of tools that the robot is required to grasp, careful consideration had to be given to the final gripper designs. An additional factor was the storage fixtures used for the tools. Feedback from stakeholders indicated that the existing tool holders were not to be modified, making them a key design constraint. Although compliant gripper systems can handle a broad range of object geometries, the conditions imposed by the pegboard storage arrangement made the development of a truly universal end effector impractical. As a result, the team elected to develop a system that allows the robot to automatically exchange end effectors according to the tool being handled, thereby enabling more reliable and optimised pickup performance across all tool types.

A prototype end effector was developed, which was a rack-and-pinion gripper mechanism with swappable fingers. This design was chosen primarily due to being straightforward to implement as shown in the figure 29 for the concept screening, while also aligning with the direction validated during the interim presentation. The prototype gripper can be seen in figure 30.

Non linear dynamic analysis were also performed on the compliant gripper fingers as seen in the figure 31 to evaluate the deformations when in operation. These results were then validated during grasping tests with various tools.

Concepts were also evaluated for the tool changer mechanism as shown in figure 32, with the utilisation of electromagnets being selected as the approach. Tests were conducted to validate the electromagnet strength with a selected steel plate of certain thickness and were successfully completed as shown in the figure 33. The exact calculations for the thickness of steel is shown in Appendix E.

Following the validation of these concepts, the next sections will cover the detailed design and development of the finalised end effector and tool changer prototype systems, including the mechanical design, electrical integration, and control strategies implemented to achieve the required functionality.

3.2 End effector

3.2.1 Mechanical Development

Regarding the mechanical design of the end effector, the body was redesigned to further optimise its overall size, with particular emphasis placed on enclosing the mechanism’s dynamic components, namely the rack-and-pinion assembly. This was done to reduce the likelihood of pinch injuries should a user place their fingers near the moving regions of the mechanism during operation. Figure 34 above shows the exploded view of the gripper while figure 35 below shows the comparison of the initial prototype and the final prototype, where the size reduction and improved aesthetics of the final design can be observed.

Most critically, the gripper attachment interface was redesigned to incorporate a universal mounting feature, allowing direct attachment to the slave side of the tool changer. This modification was essential in enabling automatic end effector exchange within the overall system. Further details of this interface are presented in Section 3.3.1.

The method of swapping gripper fingers was also improved, such that each finger could be secured to the rack using only two M3 × 25 mm screws as seen in figure 36. This reduced the complexity of finger replacement and improved maintainability during testing.

Following successful preliminary evaluation of the compliant gripper finger design, this concept was retained in the updated iteration. Experimental testing showed that the compliant fingers were able to successfully grasp screwdrivers, wrenches, and pliers. However, due to the mounting geometry and handling requirements of the mallet, a different gripper finger configuration was required to achieve reliable pickup. For this tool, the selected approach involved lifting the mallet while maintaining its orientation throughout the transfer process until deposition, thereby ensuring stable handling. The different gripper options are shown in figure 37.

3D Model of the Gripper

3.2.2 Electrical Development

For the electrical subsystem, the DS3218MG servo motor was selected primarily for its suitable torque characteristics for the intended gripping application. As the servo has a maximum rated input voltage of 6.4 V, an LM2596 buck converter was incorporated to step down the available 12 V supply to an appropriate operating voltage. At the same time, retaining a 12 V supply rail preserves the option of integrating higher-torque actuators in future revisions, such as the Dynamixel AX-12A, should greater actuation capability be required.

To control the opening and closing motions of the gripper, an STM32F103C8T6 microcontroller was used. The tool-side circuit board, which formed part of the tool changer electronics, was designed to house the control circuitry and embedded firmware responsible for end effector actuation. Further details of this implementation are provided in the following section.

Figure 38 shows the electrical block diagram of the gripper system.

3.2.3 Integration and Testing

The integration of the electrical subsystem with the mechanical components was a critical phase in the development process. This involved ensuring that all electrical connections were properly made and that the control circuitry functioned as intended. Initial testing focused on validating the functionality of the gripper mechanism, including the opening and closing motions on the specified tools for pickup. Figure 39 above showcases the grippers undergoing trials. The videos below show the testing of the gripper with the compliant fingers and the mallet configuration, demonstrating successful grasping and handling of the tools.

In terms of pick reliability and repeatability, the gripper was able to achieve consistent performance across multiple trials with the compliant fingers for screwdrivers. The mallet configuration also showed reliable handling, although the specific geometry of the mallet required careful positioning to ensure a secure grip. Overall, the integrated end effector system met the functional requirements for tool handling within the intended application context.

However, for the compliant fingers, the gripper was not able to achieve a reliable grasp on the screwdrivers when the gripper is offset from the ideal grasping position. This is because the compliant fingers rely on deformation to conform to the tool geometry, and significant offsets can lead to insufficient contact and grip strength. It is because of this that the computer vision subsystem was implemented to provide feedback on the gripper position relative to the target tool, allowing for adjustments to be made to ensure a successful grasp. Future iterations of the gripper design could also explore modifications to the finger geometry or material properties to improve robustness against positional offsets. Figure 40 shows some examples of failed grasps.

3.3 Tool Changer

3.3.1 Electrical Development

As the electrical subsystem of the tool changer imposed several constraints on the overall mechanical implementation, it is addressed first in this section.

Evaluation of the available computing hardware, particularly the UNO-238 V2 Industrial Mini PC, showed that the accessible interfaces were limited to 3.3 V GPIO logic and CAN Bus 2.0B. CAN Bus was selected over conventional serial communication because its differential signalling offers greater robustness against electromagnetic interference. This was especially important as communication lines had to pass through the robot arm structure, where the cobot’s AC servo drives could introduce electrical noise.

Based on the available system peripherals, the power requirements of the end effector prototype, and the results obtained from preliminary testing of the 24 V electromagnet, a set of design deliverables was established for the control board.

To further reduce bill-of-materials cost, the board was designed to operate as either a master or slave through configuration. In the master role, it is mounted on the cobot side to handle power delivery and control the tool changer release mechanism. In the slave role, it is mounted on the end effector side to receive system commands and control the actuators and sensors of the attached tool. This reduced the number of unique board designs required and simplified manufacturing and integration.

After evaluating the range of microcontrollers available on the market, the STM32F103 was selected because it satisfied the functional requirements of the system while maintaining relatively low cost and manageable design complexity. The device provides sufficient digital I/O and integrated peripherals for the intended application, including CAN communication, UART communication, timer functions, and general-purpose control signals.

Based on the system requirements, an interaction architecture shown in figure 41 was then defined between the control boards and the Mini PC, such that the Mini PC serves as the primary computation and supervisory control unit, while the control boards handle local peripheral interfacing, power switching, and actuator-level control.

For the CAN Bus communication circuit, the SN65HVD232 (Figure 42) transceiver was selected due to its compatibility with 3.3 V microcontroller logic and its suitability for direct integration with the STM32-based control architecture. In addition, the device is widely used in embedded CAN applications and is supported by substantial reference material and implementation examples, which reduced development risk and simplified hardware validation.

Following the interim presentation, the selected electromagnet required a 24 V supply, while the gripper subsystem only required 12 V operation. As the STM32F103 operates using 3.3 V logic, dedicated switching circuits were required to allow the low-voltage control signals from the microcontroller to safely and reliably switch the higher-voltage loads.

For the 24 V electromagnet, a low-side switching circuit based on an N-channel MOSFET was implemented as shown in figure 43. In this topology, the MOSFET is positioned between the load and ground, such that enabling the MOSFET completes the current path through the electromagnet. Since the STM32 microcontroller operates at 3.3 V logic, its GPIO output is insufficient to directly provide the higher gate drive voltage required to fully enhance the selected N-channel MOSFET. A transistor driver stage consisting of an NPN transistor and a PNP transistor was therefore used as a level-shifting interface.

The 3.3 V GPIO signal first drives the NPN transistor, which then controls the PNP transistor connected to the 12 V rail. When activated, the PNP transistor pulls the MOSFET gate toward 12 V, thereby providing a substantially higher gate-to-source voltage than could be achieved directly from the microcontroller output. This ensures full enhancement of the MOSFET, resulting in a lower R_DS(on) and reduced power dissipation. A flyback diode was connected across the electromagnet to clamp the reverse voltage generated during turn-off, protecting the switching device against inductive overvoltage.

For switching the 12 V supply to the end effector, a high-side switching circuit based on a P-channel MOSFET was used as shown in figure 44. In this case, the MOSFET is placed on the positive supply rail, allowing the load to be disconnected from the 12 V source when required. Because the source terminal of the P-channel MOSFET sits near 12 V during operation, the microcontroller’s 3.3 V logic output alone is insufficient to directly pull the gate to the required level for effective switching. To address this, an NPN transistor was introduced to pull the MOSFET gate toward ground when commanded, producing a sufficiently negative V_GS to turn the P-channel MOSFET on. When the transistor is turned off, the gate is pulled back toward the source potential, turning the MOSFET off. This arrangement allowed the microcontroller to control the 12 V output rail safely using its 3.3 V logic levels.

To enable repeated coupling and decoupling between the robot-side and tool-side control boards, a pogo-pin contact interface was adopted. Spring-loaded pogo pins such as the ones shown in figure 45 below were placed on the tool side, with matching pogo pads on the robot side. Appendix F10 and F11 provide the specifications of the pogo pins used.

From these requirements, a hardware block diagram seen in figure 46 was developed to define the major functional sections of the circuit board and to map the selected components to each subsystem requirement.

Considering the tight dimensional constraints imposed by the mechanical design of the tool changing system, the control board dimensions were fixed at 30 mm × 40 mm × 10 mm in length, width, and height respectively. Under these constraints, the use of off-the-shelf development boards, prebuilt modules, or prototype perfboard construction was not suitable for the deployed prototype. As a result, a custom printed circuit board was designed in KiCad, based on circuit simulation, calculation, and validation carried out using LTspice and SMath Solver.

The board development was done iteratively, starting with the design and testing of individual circuit sections such as the CAN transceiver interface, the MOSFET switching circuits, and the microcontroller firmware. After validating each section independently, the full circuit was integrated into a single board design, which was then fabricated and assembled for final testing and deployment. Figure 47 shows the iterative prototypes developed during the design process, leading up to the final integrated control board design. See Appendix F13 to F16 for the prototype iterations in more detail.

The final control board integrates the STM32F103C8T6 microcontroller, the low-side and high-side switching circuits, and an onboard CAN Bus 2.0B transceiver into a single compact design. In the tool-side configuration, the board additionally supports up to six timer-based input/output channels, allowing it to control end effectors with as many as six servos. Figure 48 and Figure 49 below showcase the key components on the board. For the selection of components and calculations, see Appendix F1.

3D Model of the Tool Changer PCB

Appendix F2 to F8 goes through the board layers in more detail.

To validate the fabricated control boards, firmware was first uploaded through the onboard debugger interface, followed by LED tests to confirm basic microcontroller operation. Communication tests were then carried out to verify the CAN circuitry by transmitting signals from the robot-side board to the tool-side board, with onboard debugging LEDs used as visual indicators of successful data exchange. Finally, the MOSFET and transistor switching circuits were tested to confirm correct control of the 12 V supply and the electromagnet, both of which were successfully validated. Figure 50 below shows the testing of the control boards.

Upon completion of these electrical tests, the boards were integrated into the mechanical housings for the Robot-side and Tool-side assemblies, as seen in figure 51 below. More details on the mechanical design of the housings are provided in the following sections.

3.3.2 Mechanical Development

Regarding the mechanical design of the tool changing system, 3 key portions of the system need to be developed, namely:

- The Robot Side unit

- The End Effector Side unit

- The Tool Holder

3.3.3 The Robot Side Unit

The Robot side tool unit shown in figure 52 is meant to house the electromagnet and corresponding circuit board, with a major consideration for the alignment of the Tool side unit when docking. Successful, repeatable alignment of the tool side unit is crucial for both mechanical and electrical connections.

The docking interface was designed with chamfered alignment features to improve passive self-alignment between the robot-side and tool-side units. Tapered ends were applied to the tool-side alignment pins, while chamfered entry features were incorporated into the mating holes on the robot-side interface. The purpose of these lead-in geometries was to compensate for minor positional errors in the robot approach by guiding the mating parts into alignment during contact.

As shown in figure 53, the design reduces sensitivity to small lateral misalignments by converting initial contact forces into corrective guiding forces, allowing the interface to self-center before full engagement of the electromagnetic and electrical connection surfaces. As a result, the mechanism achieves more reliable and repeatable docking, while also reducing the risk of collision, jamming, and localized wear at the mating interface. The selected chamfer geometry was sufficient to tolerate approximately ±1 mm of positioning error during insertion. Figure 54 shows the free body diagram of the alignment pins working, where the forces generated from the chamfered features guide the pins into the holes for proper alignment.

The mounting of the camera was done on the robot side for the detection of the tools and assisting in grabbing. Further elaboration on the exact software is shown in section 4. In terms of the mechanical mounting, consideration was given to the camera’s RGB and Depth FOV as seen in figure 55, to ensure that the camera image was not blocked when the Tool side with the end effector was docked together.

3.3.4 The End Effector Side Unit

For the tool side unit shown in figure 56, the primary design considerations involve the mounting of the corresponding steel plate, of which the validation was done from the interim presentation period, with the calculations seen in appendix E alongside the placement of the dowel pins. The corresponding tool changer circuit system is mounted here as well.

Figure 57 shows the mounting holes that are placed at the underside of the holder for customisability in gripper placements and options. Additionally, the edges of the holder are designed to double function as the supports on the tool holder.

3D Model of the Tool Changer Unit

3D Model of the Tool Changer Unit with end effector attached

3.3.5 The Tool Holder

The tool changer holder shown in figure 58 was created with the following considerations on the robot's physical approach for docking and undocking alongside the part complexity.

The tool pickup strategy was intentionally designed as a vertical lift followed by linear forward clearance, rather than using an angled holder. This approach decouples load transfer from disengagement, resulting in more predictable tool extraction, reduced sideloading, improved repeatability, and simpler tolerance management. While an angled holder may appear to assist release geometrically, it would also introduce greater dependence on precise alignment and could increase the risk of rubbing, wedging, and inconsistent extraction. The selected method therefore offered a more robust and controllable solution for reliable autonomous tool exchange.

In this application, robustness and repeatability were prioritized over passive geometric assistance. The chosen holder geometry ensures the robot executes a deterministic extraction sequence that is easier to validate and better suited to autonomous operation. The docking and undocking sequence is shown in more detail in the figures 59 and 60 below, where the vertical lift and forward clearance can be observed.

The retaining “fingers” of the tool holder were intentionally designed to constrain the tool in the X and Y directions. This was necessary because the end effector behaves as a separate rigid body with respect to the main robotic platform. During AMR motion, especially under sharp angular turning, inertial loading acting on the gripper can generate lateral displacement relative to the holder. Without sufficient constraint, this could result in unintended disengagement of the tool from its storage interface. The addition of X- and Y-axis constraint therefore improves retention security during dynamic operation as shown in figure 61.

The basket unit as seen in figure 62 was designed to be mounted to the tool holder body, to allow for modularity should the basket be damaged or require replacement. The basket was designed to be large enough to accommodate the tools while also being compact to minimise the overall footprint of the storage system.

3D Model of the full Tool Changer Assembly

3.4 Overall integration and testing of tool changer and end effector

After the individual development of the end effector and tool changer subsystems, the final phase of this section involved the integration of all components into a cohesive system as shown in figure 63, followed by comprehensive testing to validate functionality and performance. This process included mechanical assembly, electrical wiring, firmware deployment, and system-level validation through a series of test scenarios designed to evaluate the reliability and effectiveness of the tool changing mechanism under realistic operating conditions.

The tool changing system was subjected to a range of tests, including repeated docking and undocking cycles to assess the durability and reliability of the mechanical interfaces, as well as functional tests to confirm the correct operation of the electrical control circuits and the successful actuation of the gripper mechanism as shown in figure 64 below.

The videos below demonstrate key aspects of the integrated system in action, showcasing the tool changer’s ability to autonomously exchange end effectors and transmit data. The video showcasing CAN Bus transmission code is in section 5, under the software control stack. Over 300 docking/undocking cycles were completed and only 2 failures are recorded, due to firmware bugs that were subsequently fixed.

The videos below show the testing of the toolchanging system working to perform a pick and place operation. This test was hardcoded with no feedback data.

When referencing the specification table on the size, mass and pick capabilities of the system, the system meets the requirements as follows:

While most of the requirements were met, the system was not able to meet the payload support requirement for both the gripper and tool changer subsystem. The gripper was only able to reliably grasp up to 1kg of payload, while the tool changer was only able to support up to 5kg of payload before failure. This was due to limitations in the strength of the electromagnet used in the tool changer, as well as limitations in the mechanical design of the gripper and tool holder. Future iterations of the design could explore stronger electromagnets, improved gripper designs, and more robust tool holder geometries to increase the maximum payload capacity of the system.

5 Software Control Stack

5.1 Updated software strategy and architecture

From the interim presentation, a user sequence diagram, shown in figure 100, and an initial software architecture diagram, shown in Figure 101, were developed to represent the idealised control flow and software structure of the system.

For the final deployed prototype, the software architecture was further streamlined, as illustrated in the updated diagram shown below in figure 102.

Throughout the development of the final prototype software, the implementation approach was continually refined and, where necessary, restructured to meet the practical operating requirements of the system. The following sections describe in detail the individual nodes and their respective functionalities that enable the overall system to operate.

5.2 Moveit2 Trajectory Planner

The MoveIt2 trajectory planning node is responsible for generating feasible manipulator motion trajectories for the robotic arm. Its purpose is to translate target poses or motion goals into executable arm movements that respect the robot’s kinematic structure, joint limits, and safety constraints. In the developed system, this node is essential for actions such as moving the Cobot to predefined standby positions, approaching a tool holder, aligning with an object, executing pickup or placement motions, and returning the arm to a safe home configuration.

One of the main reasons for using MoveIt2 is its ability to handle inverse kinematics and motion planning in a standardized framework. In a practical system, commanding the robot to move from one pose to another is not simply a matter of assigning Cartesian coordinates. The software must determine whether the target can be reached, which joint configuration should be used, and whether the resulting motion is smooth and safe. MoveIt2 provides this capability while also supporting collision checking, trajectory generation, and planning scene integration when required. This significantly reduces the need for manual low-level trajectory derivation and improves the reliability of arm motion.

From the interim presentation, various trajectory planners were tested and the PILZ trajectory planner was selected for stable trajectory planning and ability for linear trajectory application.

The diagram in figure 103 shows how the node, called the Master Cobot Action Node, interacts with the other software subsystems in the overall architecture. The node’s endpoints feature Ros2 services that allow for the control and data reading of the cobot during its execution. The sequencer node, the node that handles the direct systems, mainly interacts via this service calls which are sequence requests.

5.3 AMR control stack

The SESTO node is responsible for managing communication with the SESTO Magnus Lite AMR platform and providing the rest of the software system with an interface for navigation-related and control functions. Its primary function is to accept location-based commands from the sequencer and convert them into navigation requests that can be understood by the mobile platform. This involves calling external APIs, sending requests to the base controller, or continuously polling platform status to monitor progress.

From the diagram above in figure 104, the main interactions with the software architecture involve the SESTO node accepting service and action requests from the sequencer. These service cand action request are directly tied to API calls that are done to the SESTO’s onboard API server to execute the motions. The table below shows the endpoints utilised for the system.

The SESTO node also contributes to execution reliability by returning status information back to the sequencer. Once a navigation goal is issued, the sequencer does not simply assume the movement has completed. Instead, it relies on the SESTO node to monitor whether the platform is travelling, has arrived, or has encountered an issue. This is especially important in integrated workflows where downstream actions, such as tool pickup or delivery, should only begin once the base is correctly positioned. Without this feedback mechanism, the system could attempt manipulator operations at the wrong location, leading to failed or unsafe execution.

When the AMR reaches its destination, it will enter in Station-Lock mode, where no navigation requests will be accepted until specifically released. This is used as an intermediary safety layer to prevent any navigation requests from being submitted which may lead to unsafe conditions if the arm also were to be moving simultaneously. The release of this Station-Lock is done via the API endpoint /amrs/release/station and is shown in figure 105.

For the creation of new waypoints, this can also be done using API calls to the system but for convenience, the Fleet Server UI seen in figure 106, onboard the AMR was utilised as the GUI overlay shows the location of the endpoints, leading to easier implementation. Each key point can then be referenced in the API calls.

The navigation accuracy is noted to be a point of concern, repeatability tests were done which shows that after 50 navigation attempts to the same defined location, the AMR can drift about +- 20mm in both X and Y directions. This poses an issue during the pick and place as hardcoded motions will not suffice to grab the items as the whole system can drift far enough that the gripper will miss outright. This inaccuracy issue is solved via the utilisation of computer vision as mentioned in section 4 and in section 5.4.

5.3.1 SESTO Navigation Accuracy and Mitigation

The SESTO Magnus Lite AMR platform exhibits a navigation accuracy issue, with observed drift of approximately ±20 mm in both X and Y directions after repeated navigation attempts to the same location. An example of this drift is shown in figure 107 where the AMR is seen to have deviated from a marker reference. This level of inaccuracy poses a significant challenge for tasks such as pick-and-place operations, where precise positioning is critical for successful item grasping.

Based on the datasheet, the Sesto Magnus Lite has a navigation accuracy of ±20 mm, which is consistent with the observed drift during testing. The exact positional readings can be found in Appendix N. This means that even if the AMR successfully reaches the general vicinity of the target location, there is still a possibility that it may be misaligned by up to 20 mm, which can lead to failed grasps if the robot arm relies solely on hardcoded motions based on expected coordinates.

Consideration was placed for the utilisation of a docking marker that the AMR can reference and dock to. However, the placement of the docking marker and low profile design makes it a tripping hazard for people as seen in figure 108. Hence this idea was scrapped.

To mitigate this issue, the system incorporates computer vision techniques to dynamically adjust the robot's position during pick-and-place tasks. By using visual feedback from the camera, the robot can detect the actual location of the target item and make real-time adjustments to its position, compensating for any drift caused by navigation inaccuracies.

5.4: CAN Bus Communication

The CAN bus node provides the low-level hardware communication interface between the ROS2 software stack, and the physical end-effector or peripheral devices connected over the CAN network. The CAN bus node interfaces with the drivers and corresponding CAN peripherals onboard the UNO 238 V2 MiniPC, whose interfaces are shown in figure 109.

In the context of the robotics system, the CAN node mainly interfaces with the tool changer unit circuits boards, seen in section 3. The interaction between the CAN node and the peripherals can then be illustrated in figure 110 below:

The CAN Node itself opens various services that the sequencer can call. These services are for the control of the tool changer’s electromagnet, power delivery and the gripper’s open and close command. Upon calling any of these services, the CAN Node transmits a CAN frame that can be seen in figure 111.

In this CAN frame, the commands for the electromagnet, power delivery and gripper state are seen in the bit 6, 5 and 4 fields whilst bit 3 to 0 are reserved for other command structures.

Whilst the Tool side control board can also receive CAN frames from the MiniPC, a separated form of control was done instead, with the Robot side control board managing the commands from the MiniPC and monitors the state of the Tool side control board. This is so that in the event of any loss of connection on the tool side, the Robot side can notify the MiniPC.

5.5: Computer Vision

The computer vision node is responsible for processing tool recognition data and returning the depth and detection information required for the system to perform pick-and-place operations or to terminate an order when the requested tool is not detected. The detailed processing methodology is discussed in Section 4, Computer Vision. Accordingly, this section focuses on the role of the computer vision node within the control architecture and its interactions with the rest of the software stack.

The interaction diagram in figure 112 illustrates the relationship between the computer vision node and the wider control stack, with the sequencer node acting as the primary service client.

Upon receiving a request from the sequencer node, the computer vision node takes in the name of the requested tool and begins processing the visual input to determine whether the tool is present. If the request is completed successfully and valid frame data is returned, the sequencer node then computes the new target coordinates required for motion execution. While this process is based on frame transformations, it was simplified into vector calculations under the assumption of fixed frame relationships. The following sections describe this calculation in greater detail.

5.5.1 Target Position Derivation

To command the robot such that the gripper reaches the target detected by the depth camera, a vector-based derivation was used. Under the simplifying assumption that all coordinate frames are aligned, rotational transformations reduce to the identity and only translational offsets need to be considered. The required position can therefore be obtained through direct vector addition and subtraction. Figure 113 illustrates the relevant coordinate frames and vectors involved in the derivation.

5.5.2 Symbol Definitions

5.5.3 Derivation

The target position in the world frame is first expressed by summing the current endpoint position, the camera offset from the endpoint, and the detected target offset from the camera:

To ensure that the gripper coincides with the detected target, the desired gripper position is set equal to the target position:

Since the gripper is mounted with a known offset relative to the robot endpoint, the desired gripper position may also be written as:

Substituting Equation (2) into Equation (3) gives:

Rearranging for the desired robot endpoint position:

Substituting Equation (1) into Equation (5) yields the final expression for the desired endpoint position:

5.5.4 Final Expression

Equation (6) shows that the desired robot endpoint position is obtained by starting from the current endpoint position, adding the camera mounting offset, adding the target offset measured by the depth camera, and subtracting the gripper offset. The gripper term is subtracted because the robot is controlled through the endpoint frame, whereas the actual point required to coincide with the target is the gripper.

After the calculation is completed, the sequencer then moves the end effector into the desired position by sending direct frame goal requests to the MoveIt2 trajectory planner, which then computes the required trajectory to move the robot to the desired position.