Hi Folks, In this post, we will explore the basics of using MediaPipe on the Android platform. This post is for complete Beginners, will cover basics, how it works, how to create APK or ARR.
What Mediapipe is?
MediaPipe is a powerful cross-platform framework for building multimedia processing pipelines. It provides a set of tools and libraries for building real-time applications for processing video, audio, and other types of media.
Why Mediapipe?
- Open-Source
- Cross-Platform
- High Performance
Mediapipe Components ?
MediaPipe is made up of several components that work together to enable the creation of multimedia processing pipelines. These components include:
Nodes: Nodes are the building blocks of MediaPipe pipelines. They perform specific processing tasks, such as video and audio decoding, image processing, and machine learning inferencing.
Graphs: A graph defines the processing steps for a MediaPipe pipeline. It consists of nodes connected in a specific order, and the input and output data flows through the graph, passing from node to node.
Calculator Framework: The Calculator Framework is a component of MediaPipe that provides the infrastructure for defining, executing, and managing pipelines. It provides a simple interface for connecting nodes and managing the flow of data through the pipeline.
Calculator Contracts: Calculator Contracts are the interface definitions that specify the input and output requirements for each node in a MediaPipe pipeline.
Protobuf: Protobuf is a data format used by MediaPipe to serialize and deserialize data. It is used to pass data between nodes in a pipeline and to persist pipelines and their configurations.
Tools and Libraries: MediaPipe provides a set of tools and libraries for building pipelines, including pre-built nodes for common processing tasks and libraries for Machine learning and Computer vision.
Hmm, Hope you got some idea about what mediapipe is!
- Mediapipe has two major parts. One is Solutions and another one is Framework.
Solutions are their Pre-build Library files, which we can use it in Android Studio via Gradle Implementation [See more info about here.]. We can use Facemesh, Hands, Face Detection currently with Pre-Build Lib Files. It is pretty much easy, Just Go → download this files and run in android studio.
As of Now, Solutions API in ALPHA, but it’s works fine.
Framework is nothing but we can customize their graph, calculator file and we can create APK or Android AAR lib file from their Framework.
To Build APK or AAR we are using tool called Bazel.
Setup Mediapipe in your System
You can follow their official Steps mentioned here to Setup in your system.
Note : FOR WINDOWS USER, USE WSL ONLY. [We can’t Build APK or AAR in Windows Native. So we will use WSL to build APK or AAR]
SETUP PROCESS WILL TAKE MORE TIME AND ITS NEED ATLEAST 2–3 GB OF INTERNET. [If we use WSL we have to install SDK and NDK again in the WSL Ubuntu.]
IN THIS BLOG WE GOING TO SEE ONLY ABOUT FRAMEWORK, SOLUTIONS(prebuild) IS SIMPLE,CHECK THEIR SAMPLE PROJECT.
BUILD Your First APK
Lets see how to build Hand Tracking Example.
- Install Bazelisk(It’s a user friendly launcher to install Bazel tool)
- Download Mediapipe Files from thier Github and Navigate to this path “./mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu”. You can see BUILD file, this Build File Used to Generate APK or AARfile.
Don’t worry if you can’t understand what that BUILD file consists of. Let me give some info,
- cc_binary in Bazel is a rule for compiling and linking C++ code into an executable binary. In the context of MediaPipe, it is used to build the C++ components of the framework into standalone executables that can be run on the command line.
- cc_library in Bazel is a rule for compiling and linking C++ code into a static library. In the context of MediaPipe, it is used to build the C++ components of the framework into libraries that can be linked into other targets, such as executable binaries or other libraries.
- Android_binary in Bazel is a rule for building Android applications. In the context of MediaPipe, it is used to build MediaPipe applications for the Android platform .It specifies the Android Manifest file, resources, and dependencies that make up the application, as well as any additional build options that should be used when building the application.
The application can depend on one or more cc_library or android_library targets, which will be built and included in the APK (Android Package) file whenever the android_binary target is built.
3. Now let’s Run the below command to build Hand Tracking APK.
NOTE: you need to change — config=android_arm64 this according to your need.
As soon as you run this command, it will take some amount of time, If it is First time it might be take nearly 2 hours to compile and generate the APK.
Once Generated you will get the path of APK, Install the apk!!
[For WSL Users, we can’t navigate into that folder path, so we have to copy in another folder and we can use it.]
- Create a “test” Directory,
- Copy Apk into test directory.
That’s it now install and test it.
How can we build Android Archive (AAR) ?
- Create Folder and BUILD file like below.
“mediapipe\examples\android\src\java\com\google\mediapipe\apps\aar_example\hand\BUILD”
- add below contents in BUILD file,.
Note : If you want to use another usecase like Facemesh, you have to find the right calculator for that particular usecase.
How can we find the proper Calculator ?
- Its simple, navigate to your usecase folder, for example handtracking sample folder inside the mediapipe\examples\android\src\java\com\google\mediapipe\apps folder.
- Open the BUILD file inside that folder, and you will see cc_binary
- In that we have our calculator name inside the deps
- You have to only focus on graphs.
- FYI : The second one is mediapipe JNI library[.so], it will automatically create and included in our AAR.
If it’s AAR, you have to customize the Calculators, graphs files, otherwise you will get the Drawings in the output[using OpenCV] like APK.
That’s it !!
Happy Coding.
