(Source: Itsanan/Shutterstock.com)
One of the crucial technologies underpinning Industry 4.0 is the ability to implement machine-learning inference at the edge. Depending on the application’s needs, the machine-learning inference could analyze telemetry to model predictive maintenance to prevent a line-stop situation. Alternatively, the application could inspect manufactured or packaged items on the production line.
The ability to visually inspect items on the production line requires a high-performance system capable of running image processing and machine learning algorithms at a high frame rate (>30fps frames per second). Being able to implement a solution capable of such processing also requires significant power. This is where products such as the Xilinx® Kria™ K26 System-on-Module (SoM) and KV260 Vision AI Starter Kit from Xilinx can help.
Xilinx® Kria™ K26 SoM and its KV260 Vision AI Starter Kit enable the rapid prototyping of vision and AI algorithms. The Kria K26 SoM provides design engineers with a high-performance heterogeneous system that combines both high-performance Arm® processors and advanced programmable logic. This combination allows the applications to be optimally implemented in either the processing system or programmable logic. The result of this fusion is an edge-based solution, which offers a responsive, deterministic, and power-efficient solution.
Kria differs from traditional Xilinx offerings in that it is delivered as a SoM. A SoM combines not only the integrated circuit, but also includes the necessary supporting volatile and non-volatile memories, clocks, and power supplies. The Kria K26 SoM consists of the SoC (XCK26) along with 4GB DDR4 Memory, 16GB eMMC, 512Mb QSPI, TPM Security module, and the necessary power infrastructure (Figure 1). Dual 240-pin connectors, which break out 245 I/Os, make it easy to interface with your application.
Figure 1: The Xilinx Kria K26 SoM allows design engineers to leverage the parallel nature of programmable logic combined with high-performance Arm® processor cores. (Source: Mouser Electronics)
To help design engineers get started and hit the ground running, Xilinx offers the Kria KV260 Vision AI Starter Kit. The Kria KV260 Vision AI Starter Kit includes a carrier card for the SoM, which provides the following interfaces:
These interfaces enable design engineers to create complex vision-based AI solutions. The solutions can support a range of video sources and sinks from MIPI to USB cameras and the Ethernet Real-Time Streaming Protocol (RTSP) and traditional HDMI and DisplayPort sinks.
This starter kit also comes with a range of applications that show how easy it is to get started developing vision-based AI applications. These applications include smart cameras that can detect faces; multi-stream tracking and identification; defect detection; and natural-language processing. One of the nice things about the Kria out-of-the-box architecture is the ability to recompile a different network and replace the example network.
Design engineers can use Vitis™ AI to leverage commonly used AI development frameworks such as Caffe, TensorFlow, and Pytorch. Vitis AI enables the acceleration of AI inference algorithms at both the edge and the cloud. The Vitis AI technology stack supports the commonly used frameworks and provides everything needed to develop and deploy ML/AI algorithms on the Xilinx devices, including the Kria K26 SoM.
At the heart of the Vitis AI stack is the Xilinx Deep Learning Processor Unit (DPU), which is implemented in the programmable logic and is optimized for the implementation of Convolution Neural Networks (Figure 2). It can be used to implement networks such as VGG, ResNet, GoogLeNet, YOLO, SSD, MobileNet, and FPN.
Figure 2: The Vitis AI Stack workflow offers a process to deploy deep-learning inference applications on Xilinx DPU. (Source: Xilinx)
To leverage the parallel nature of programmable logic, the DPU executes networks that have been quantized to int-8 using the AI Quantizer.
Once the DPU model has been implemented in the hardware and the network is trained and compiled using Vitis AI, the software can be developed using Vitis to create the complete final solution (Figure 3).
Figure 3: Xilinx Vitis AI deployment model diagram (Source: Xilinx)
Let’s look deeper at how the Xilinx Kria SoM can be used for a manufacturing application. Creating a manufacturing application does not necessarily require any programmable logic design. However, it will require software development and the ability to train and compile a new machine-learning model using Vitis AI from Xilinx.
The Kria K26 SoM and KV260 Vision Starter Kit are perfect for applications where fast image processing is required, such as detecting whether or not a label has been correctly applied to a shipping box on the production line. In this example, the design engineer uses the Kria K26 SoM to inspect packages on the production line and correctly identify the location of a label on a box through the Mobilenet network. This network can be trained from a dataset of images containing both correct and incorrect label locations (Figure 4 and Figure 5).
Figure 4: An example of correct label locations using the Mobilenet network (Source: Mouser Electronics)
Figure 5: An example of incorrect label locations using the Mobilenet network (Source: Mouser Electronics)
Once trained, this network can then be deployed on the Kria K26 SoM, in conjunction with Linux GStreamer support, to inspect packages on a production line in real-time (Figure 6). The deployed application can take the appropriate response if it detects a mislabeled box.
Figure 6: An example of Xilinx correctly identifying a correct label (Source: Mouser Electronics)
Production lines are fast-paced environments. The ability to detect manufacturing or packaging defects before shipping is critical to improving delivery and customer satisfaction. However, automated inspection equipment needs to process and make decisions without slowing the production line. Products such as the Xilinx Kria K26 SoM and Kria KV260 Vision AI Starter Kit can help.
Design engineers can use the Kria KV260 Vision AI Starter Kit to quickly start developing vision and AI prototyping solutions and demonstrating proof of concepts. The Kria K26 SoM enables the portability of the design from concept, prototyping to the final design solution with minimal change required, except perhaps to the interface with the carrier card.
The development cost and risk associated with creating an embedded system solution with complex heterogeneous FPGAs, memories, and power architectures can be mitigated using the Kria K26 SoM. By combining the Vitis AI stack and out-of-box solutions, the Kria K26 SoM enables the development team to create complex vision and AI-based solutions by exploiting the high-performance nature of programmable logic without the need to be a programmable logic design specialist.
If you would like to learn more about the Kria SOM and view a step-by-step walkthrough of creating an AI solution for industrial applications, see the Deploying Edge-Based AI Using the KRIA SoM project article.
Adam Taylor is a professor of embedded systems, engineering leader, and world-recognized expert in FPGA/System on Chip and Electronic Design.