Adding Image Segmentation for Android

Signed-off-by: Ravi Kumar Neti <[email protected]>

ai-solutions/android/04-ImageSegmentation/README.md

@@ -0,0 +1,130 @@
+# Table of Contents
+
+- [Table of Contents](#table-of-contents)
+- [Introduction](#introduction)
+    + [About "Image Segmentation"](#about-image-segmentation)
+    + [Pre-Requisites](#pre-requisites)
+- [Model Selection and DLC conversion](#model-selection-and-dlc-conversion)
+    + [Model Overview](#model-overview)
+    + [Steps to convert model to DLC](#steps-to-convert-model-to-dlc)
+- [Source Overview](#source-overview)
+    + [Source Organization](#source-organization)
+    + [Code Implementation](#code-implementation)
+- [Build APK file with Android Studio](#build-apk-file-with-android-studio)
+- [Results](#results)
+
+# Introduction
+
+### About "Image Segmentation" 
+
+- Current project is an sample Android application for AI-based Image Segmentation using [Qualcomm® Neural Processing SDK for AI](https://developer.qualcomm.com/sites/default/files/docs/snpe/index.html) framework. 
+- We have used 5 Models in this Solution
+- This sample segments objects in the Image.
+- DLC models take only fixed input size.
+- If users intend to use a different model in this demo framework, **image pre/post processing will be needed**. 
+- Current pre/post processing is specific to the models used. 
+
+### Pre-Requisites 
+
+- Qualcomm® Neural Processing SDK for AI setup should be completed by following the guide here : https://developer.qualcomm.com/sites/default/files/docs/snpe/setup.html
+- Android Studio to import sample project
+- Android NDK to build native code
+- Install opencv using ```pip install opencv-python```
+
+# Model Selection and DLC conversion
+
+### Model Overview
+
+Please refer to Models repository for model overview
+<TODO> Add public link
+
+### Steps to convert model to DLC
+Please refer to Models repository for model overview
+<TODO> Add public link
+
+# Source Overview
+
+### Source Organization
+
+- <DIR> demo: Contains demo video, GIF 
+- <DIR> app: Contains source files in standard Android app format.
+- app\src\main\assets : Contains Model binary DLC
+- app\src\main\java\com\qcom\imageSegmentation : Application java source code
+- app\src\main\cpp : Application C++(native) source code
+- sdk : Contains openCV sdk (Will be generated using _ResolveDependencies.sh_ )
+
+### Code Implementation
+
+- Model Initialization
+
+   `public boolean loadingMODELS(char runtime_var, String dlc_name)`
+  - runtime_var: Possible options are D, G, C. 
+  - dlc_name: Name of the DLC.
+
+- Running Model
+
+  - Following is the Java Function, that handles model execution. This function iternally calls sub functions to handle pre-processing and post-processing
+
+      `inferSNPE(inputMat.getNativeObjAddr(), outputMat.getNativeObjAddr())`
+       - inputMat is opencv Matrix that contains input image.
+       - outputMat is the destination for the output image
+
+   - C++ function that handles preprocessing for the input image.
+
+       `void preprocess(std::vector<float32_t> &dest_buffer, cv::Mat &img)`
+
+   - Model gives a segmented map that we can see in executeDLC function
+
+       `void executeDLC(cv::Mat &img, int orig_width, int orig_height, float &inferenceTime, cv::Mat &destmat)`
+
+   - SNPE API function that runs the network and give result 
+
+       `snpe->execute(inputMap, outputMap);`
+
+
+# Build APK file with Android Studio  
+
+1. Clone this repo.
+2. Generate DLC using the steps mentioned.
+3. Run below script, from the directory where it is present, to resolve dependencies of this project.
+
+	`bash resolveDependencies.sh`
+
+   * This script will download opencv and paste to sdk directory, to enable OpenCv for android Java.
+   * This script will copy snpe-release.aar file from $SNPE_ROOT to "snpe-release" directory in Android project.
+
+	**NOTE - If you are using SNPE version 2.11 or greater, please change following line in resolveDependencies.sh.**
+   ```
+	From: cp $SNPE_ROOT/android/snpe-release.aar snpe-release
+	To : cp $SNPE_ROOT/lib/android/snpe-release.aar snpe-release
+	```
+
+4. Import folder ImageSegmentation as a project in Android Studio 
+5. Do gradle sync
+6. Compile the project. 
+7. Output APK file should get generated : app-debug.apk
+8. Prepare the Qualcomm Innovators development kit(QIDK) to install the application (Do not run APK on emulator)
+9. Install and test application : app-debug.apk
+
+```java
+adb install -r -t app-debug.apk
+```
+
+10. launch the application
+
+Following is the basic "Image Segmentation" Android App 
+
+1. Select one of the models
+3. Select the run-time to run the model (CPU, GPU or DSP)
+4. Observe the result of model on screen
+5. Also note the performance indicator for the particular run-time in mSec
+
+Same results for the application are shown below 
+
+# Results
+
+- Demo video, and performance details as seen below:
+
+![Demo video.](demo/VisionSolution2-ImageSuperResolution.gif)
+
+###### *Qualcomm Neural Processing SDK and Snapdragon are products of Qualcomm Technologies, Inc. and/or its subsidiaries. AIMET Model Zoo is a product of Qualcomm Innovation Center, Inc.*

ai-solutions/android/04-ImageSegmentation/app/build.gradle

@@ -0,0 +1,63 @@
+apply plugin: 'com.android.application'
+
+android {
+    compileSdk 33
+    buildToolsVersion "30.0.3"
+
+    defaultConfig {
+        applicationId "com.qcom.aistack_segmentation"
+        minSdk 24
+        targetSdk 30
+        versionCode 1
+        versionName "1.0"
+
+        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
+        externalNativeBuild {
+            cmake {
+                cppFlags "-std=c++11 -frtti -fexceptions"
+                arguments "-DOpenCV_DIR=" + project(':sdk').projectDir + "/native/jni",
+                        "-DANDROID_TOOLCHAIN=clang"
+                targets "imageSegmentation"
+            }
+            ndk {
+                abiFilters 'arm64-v8a'
+            }
+        }
+    }
+
+    packagingOptions {
+        pickFirst 'lib/x86/libc++_shared.so'
+        pickFirst 'lib/x86_64/libc++_shared.so'
+        pickFirst 'lib/arm64-v8a/libc++_shared.so'
+        pickFirst 'lib/armeabi-v7a/libc++_shared.so'
+    }
+
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
+        }
+    }
+
+    compileOptions {
+        sourceCompatibility JavaVersion.VERSION_1_8
+        targetCompatibility JavaVersion.VERSION_1_8
+    }
+    ndkVersion '21.4.7075529'
+    externalNativeBuild {
+        cmake {
+            path file('src/main/cpp/CMakeLists.txt')
+        }
+    }
+}
+
+dependencies {
+    implementation fileTree(dir: 'libs', include: ['*.jar'])
+    implementation project(path: ':sdk')
+    testImplementation 'junit:junit:4.12'
+    androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.1'
+    androidTestImplementation 'com.android.support.test.espresso:espresso-contrib:3.0.1'
+    implementation 'com.android.support:design:26.0.0'
+    implementation 'com.android.support:support-v4:26.0.0'
+
+}

ai-solutions/android/04-ImageSegmentation/app/local.properties

@@ -0,0 +1,8 @@
+## This file must *NOT* be checked into Version Control Systems,
+# as it contains information specific to your local configuration.
+#
+# Location of the SDK. This is only used by Gradle.
+# For customization when using a Version Control System, please read the
+# header note.
+#Sat Jan 07 01:53:02 IST 2023
+sdk.dir=C\:\\Users\\shubgoya\\AppData\\Local\\Android\\Sdk

ai-solutions/android/04-ImageSegmentation/app/proguard-rules.pro

@@ -0,0 +1,21 @@
+# Add project specific ProGuard rules here.
+# You can control the set of applied configuration files using the
+# proguardFiles setting in build.gradle.
+#
+# For more details, see
+#   http://developer.android.com/guide/developing/tools/proguard.html
+
+# If your project uses WebView with JS, uncomment the following
+# and specify the fully qualified class name to the JavaScript interface
+# class:
+#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
+#   public *;
+#}
+
+# Uncomment this to preserve the line number information for
+# debugging stack traces.
+#-keepattributes SourceFile,LineNumberTable
+
+# If you keep the line number information, uncomment this to
+# hide the original source file name.
+#-renamesourcefileattribute SourceFile

ai-solutions/android/04-ImageSegmentation/app/src/main/AndroidManifest.xml

@@ -0,0 +1,42 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    package="com.qcom.aistack_segmentation">
+
+    <uses-permission android:name="android.permission.CAMERA" />
+    <uses-feature android:name="android.hardware.camera2" />
+    <uses-feature
+        android:name="android.hardware.camera"
+        android:required="false" />
+
+    <application
+        android:configChanges="orientation|screenSize"
+        android:extractNativeLibs="true"
+        android:allowBackup="true"
+        android:hardwareAccelerated="true"
+        android:icon="@mipmap/ic_launcher"
+        android:label="@string/app_name"
+        android:roundIcon="@mipmap/ic_launcher_round"
+        android:supportsRtl="true"
+        android:theme="@style/AppTheme">
+
+        <uses-native-library
+            android:name="libcdsprpc.so"
+            android:required="true"/>
+
+        <activity android:name="com.qcom.aistack_segmentation.HomeScreenActivity"
+            android:exported="true"
+            android:screenOrientation="portrait">
+            <intent-filter>
+                <action android:name="android.intent.action.MAIN" />
+
+                <category android:name="android.intent.category.LAUNCHER" />
+            </intent-filter>
+        </activity>
+
+        <activity android:name="com.qcom.aistack_segmentation.MainActivity"
+            android:exported="true"
+            android:screenOrientation="portrait">
+        </activity>
+    </application>
+
+</manifest>

ai-solutions/android/04-ImageSegmentation/app/src/main/assets/ReadMe.txt

@@ -0,0 +1 @@
+Generate model DLC and place here

ai-solutions/android/04-ImageSegmentation/app/src/main/cpp/CMakeLists.txt

@@ -0,0 +1,66 @@
+
+# For more information about using CMake with Android Studio, read the
+# documentation: https://d.android.com/studio/projects/add-native-code.html
+
+# Sets the minimum version of CMake required to build the native library.
+
+cmake_minimum_required(VERSION 3.18.1)
+
+# Declares and names the project.
+
+project("imageSegmentation")
+
+# Creates and names a library, sets it as either STATIC
+# or SHARED, and provides the relative paths to its source code.
+# You can define multiple libraries, and CMake builds them for you.
+# Gradle automatically packages shared libraries with your APK.
+
+###OPENCV
+#find_package(OpenCV REQUIRED)  ##FAILED, cannot find libcpufeatures.so
+#set(OpenCV_STATIC on)
+#set(OpenCV_DIR C:/Users/shubgoya/Desktop/SNPEworkspace/github_workspace/HRNET_posenet/opencv45/native/jni)
+find_package(OpenCV REQUIRED)
+#INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
+
+
+###INCLUDE_DIRECTORIES
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/inc/)
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/inc/zdl)
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/inc/hpp)
+include_directories(${CMAKE_CURRENT_SOURCE_DIR})
+
+
+add_library( # Sets the name of the library.
+        imageSegmentation
+
+             # Sets the library as a shared library.
+             SHARED
+
+             # Provides a relative path to your source file(s).
+              inference.cpp inference_helper.cpp imageSegmentation.cpp)
+
+# Searches for a specified prebuilt library and stores the path as a
+# variable. Because CMake includes system libraries in the search path by
+# default, you only need to specify the name of the public NDK library
+# you want to add. CMake verifies that the library exists before
+# completing its build.
+
+find_library( # Sets the name of the path variable.
+              log-lib
+
+              # Specifies the name of the NDK library that
+              # you want CMake to locate.
+              log )
+
+# Specifies libraries CMake should link to your target library. You
+# can link multiple libraries, such as libraries you define in this
+# build script, prebuilt third-party libraries, or system libraries.
+
+target_link_libraries( # Specifies the target library.
+        imageSegmentation
+
+                       # Links the target library to the log library
+                       # included in the NDK.
+                ${CMAKE_CURRENT_SOURCE_DIR}/../jniLibs/arm64-v8a/libSNPE.so
+
+                       ${log-lib} ${OpenCV_LIBS})