[WIP] Linux/Android native deploy (#980)

Makefile

@@ -128,6 +128,10 @@ ifeq ($(USE_OPENCL), 1)
 		LDFLAGS += -lOpenCL
 	endif
 	RUNTIME_DEP += $(OPENCL_OBJ)
+ifdef OPENCL_PATH
+        CFLAGS += -I$(OPENCL_PATH)/include
+        LDFLAGS += -L$(OPENCL_PATH)/lib
+endif
 else
 	CFLAGS += -DTVM_OPENCL_RUNTIME=0
 endif

apps/android_deploy/.gitignore

+*.iml
+.gradle
+/local.properties
+/.idea/workspace.xml
+/.idea/libraries
+.DS_Store
+/build
+/captures
+.externalNativeBuild

apps/android_deploy/README.md

+# Android TVM Demo
+
+This folder contains Android Demo app that allows us to show how to deploy model using TVM runtime api on a Android phone.
+
+You will need [JDK](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html), [Android SDK](https://developer.android.com/studio/index.html), [Android NDK](https://developer.android.com/ndk) and an Android device to use this.
+
+## Build and Installation
+
+### Build APK
+
+We use [Gradle](https://gradle.org) to build. Please follow [the installation instruction](https://gradle.org/install) for your operating system.
+
+Before you build the Android application, please refer to [TVM4J Installation Guide](https://github.com/dmlc/tvm/blob/master/jvm/README.md) and install tvm4j-core to your local maven repository. You can find tvm4j dependency declare in `app/build.gradle`. Modify it if it is necessary.
+
+```
+dependencies {
+    compile fileTree(dir: 'libs', include: ['*.jar'])
+    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
+        exclude group: 'com.android.support', module: 'support-annotations'
+    })
+    compile 'com.android.support:appcompat-v7:26.0.1'
+    compile 'com.android.support.constraint:constraint-layout:1.0.2'
+    compile 'com.android.support:design:26.0.1'
+    compile 'ml.dmlc.tvm:tvm4j-core:0.0.1-SNAPSHOT'
+    testCompile 'junit:junit:4.12'
+}
+```
+
+Application default has CPU version TVM runtime flavor and follow below instruction to setup.
+In `app/src/main/jni/make` you will find JNI Makefile config `config.mk` and copy it to `app/src/main/jni` and modify it.
+
+```bash
+cd apps/android_deploy/app/src/main/jni
+cp make/config.mk .
+```
+
+Here's a piece of example for `config.mk`.
+
+```makefile
+APP_ABI = arm64-v8a
+
+APP_PLATFORM = android-17
+
+# whether enable OpenCL during compile
+USE_OPENCL = 0
+```
+
+Now use Gradle to compile JNI, resolve Java dependencies and build the Android application together with tvm4j. Run following script to generate the apk file.
+
+```bash
+export ANDROID_HOME=[Path to your Android SDK, e.g., ~/Android/sdk]
+cd apps/android_deploy
+gradle clean build
+```
+
+In `app/build/outputs/apk` you'll find `app-release-unsigned.apk`, use `dev_tools/gen_keystore.sh` to generate a signature and use `dev_tools/sign_apk.sh` to get the signed apk file `app/build/outputs/apk/tvmdemo-release.apk`.
+
+Upload `tvmdemo-release.apk` to your Android device and install it.
+
+### Build with OpenCL
+
+Application does not link with OpenCL library unless you configure it to. Modify JNI Makefile config `app/src/main/jni` with proper target OpenCL configuration.
+
+Here's a piece of example for `config.mk`.
+
+```makefile
+APP_ABI = arm64-v8a
+ 
+APP_PLATFORM = android-17
+ 
+# whether enable OpenCL during compile
+USE_OPENCL = 1
+ 
+# the additional include headers you want to add, e.g., SDK_PATH/adrenosdk/Development/Inc
+ADD_C_INCLUDES = /opt/adrenosdk-osx/Development/Inc
+ 
+# the additional link libs you want to add, e.g., ANDROID_LIB_PATH/libOpenCL.so
+ADD_LDLIBS = libOpenCL.so
+```
+
+Note that you should specify the correct GPU development headers for your android device. Run `adb shell dumpsys | grep GLES` to find out what GPU your android device uses. It is very likely the library (libOpenCL.so) is already present on the mobile device. For instance, I found it under `/system/vendor/lib64`. You can do `adb pull /system/vendor/lib64/libOpenCL.so ./` to get the file to your desktop.
+
+After you setup the `config.mk`, follow the instructions in [Build APK](#buildapk) to build the Android package with OpenCL flavor.
+
+## Cross Compile and Run on Android Devices
+
+### Architecture and Android Standalone Toolchain
+
+In order to cross compile a shared library (.so) for your android device, you have to know the target triple for the device. (Refer to [Cross-compilation using Clang](https://clang.llvm.org/docs/CrossCompilation.html) for more information). Run `adb shell cat /proc/cpuinfo` to list the device's CPU information.
+
+Now use NDK to generate standalone toolchain for your device. For my test device, I use following command.
+
+```bash
+cd /opt/android-ndk/build/tools/
+./make-standalone-toolchain.sh --platform=android-24 --use-llvm --arch=arm64 --install-dir=/opt/android-toolchain-arm64
+```
+
+If everything goes well, you will find compile tools in `/opt/android-toolchain-arm64/bin`. For example, `bin/aarch64-linux-android-g++` can be used to compile C++ source codes and create shared libraries for arm64 Android devices.
+
+### Place compiled model on Android application assets folder
+
+Follow instruction to get compiled version model for android target [here.](https://github.com/dmlc/tvm/blob/master/docs/how_to/deploy_android.md#build-model-for-android-target)
+
+Copied these compiled model deploy_lib.so, deploy_graph.json and deploy_param.params to apps/android_deploy/app/src/main/assets/ and modify TVM flavor changes on [java](https://github.com/dmlc/tvm/blob/master/apps/android_deploy/app/src/main/java/ml/dmlc/tvm/android/demo/MainActivity.java#L81)
+
+`CPU Verison flavor`
+```
+    private static final boolean EXE_GPU            = false;
+```
+
+`OpenCL Verison flavor`
+```
+    private static final boolean EXE_GPU            = true;
+```
+
+
+Install compiled android application on phone and enjoy the image classifier demo using extraction model
+
+You can define your own TVM operators and deploy via this demo application on your Android device to find the most optimized TVM schedule.

apps/android_deploy/app/.gitignore

+/build

apps/android_deploy/app/build.gradle

+// import DownloadModels task
+project.ext.ASSET_DIR = projectDir.toString() + '/src/main/assets'
+project.ext.TMP_DIR   = project.buildDir.toString() + '/downloads'
+
+// Download default models(darknet framework extraction model compiled version);
+// if you wish to use your own models then place them in the "assets" directory 
+// and comment out this line.
+apply from: "download-models.gradle"
+
+apply plugin: 'com.android.application'
+
+task buildJni(type: Exec, description: 'Build JNI libs') {
+    commandLine 'sh', 'src/main/jni/build.sh'
+}
+
+tasks.withType(JavaCompile) {
+    compileTask -> compileTask.dependsOn buildJni
+}
+
+android {
+    compileSdkVersion 26
+    buildToolsVersion "26.0.1"
+    defaultConfig {
+        applicationId "ml.dmlc.tvm.android.demo"
+        minSdkVersion 17
+        targetSdkVersion 26
+        versionCode 1
+        versionName "1.0"
+        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
+    }
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
+        }
+    }
+    sourceSets {
+        main {
+            jni.srcDirs = []
+            jniLibs.srcDirs = ['src/main/libs']
+            assets.srcDirs = [project.ext.ASSET_DIR]
+        }
+    }
+}
+
+dependencies {
+    compile fileTree(dir: 'libs', include: ['*.jar'])
+    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
+        exclude group: 'com.android.support', module: 'support-annotations'
+    })
+    compile 'com.android.support:appcompat-v7:26.0.1'
+    compile 'com.android.support.constraint:constraint-layout:1.0.2'
+    compile 'com.android.support:design:26.0.1'
+    compile 'ml.dmlc.tvm:tvm4j-core:0.0.1-SNAPSHOT'
+    testCompile 'junit:junit:4.12'
+}

apps/android_deploy/app/download-models.gradle

+/*
+ * download-models.gradle
+ *     Downloads model files from ${MODEL_URL} into application's asset folder
+ * Input:
+ *     project.ext.TMP_DIR: absolute path to hold downloaded zip files
+ *     project.ext.ASSET_DIR: absolute path to save unzipped model files
+ * Output:
+ *     3 model files will be downloaded into given folder of ext.ASSET_DIR
+ */
+// hard coded model files
+def models = ['extraction.zip']
+
+// Root URL for model archives
+def MODEL_URL = 'https://github.com/PariksheetPinjari909/TVM_models/blob/master/extraction_model'
+buildscript {
+    repositories {
+        jcenter()
+    }
+    dependencies {
+        classpath 'de.undercouch:gradle-download-task:3.2.0'
+    }
+}
+
+import de.undercouch.gradle.tasks.download.Download
+task downloadFile(type: Download){
+    for (f in models) {
+        src "${MODEL_URL}/" + f + "?raw=true"
+        dest new File(project.ext.TMP_DIR + "/" + f)
+    }
+    overwrite true
+}
+
+task extractModels(type: Copy) {
+    def needDownload = false
+    for (f in models) {
+        def localFile = f.split("/")[-1]
+        if (!(new File(project.ext.TMP_DIR + '/' + localFile)).exists()) {
+            needDownload = true
+        }
+    }
+
+    if (needDownload) {
+        dependsOn downloadFile
+    }
+
+    for (f in models) {
+        def localFile = f.split("/")[-1]
+        from zipTree(project.ext.TMP_DIR + '/' + localFile)
+    }
+
+    into file(project.ext.ASSET_DIR)
+    fileMode  0644
+    exclude '**/LICENSE'
+}
+
+tasks.whenTaskAdded { task ->
+    if (task.name == 'assembleDebug') {
+        task.dependsOn 'extractModels'
+    }
+    if (task.name == 'assembleRelease') {
+        task.dependsOn 'extractModels'
+    }
+}
+

apps/android_deploy/app/src/main/AndroidManifest.xml

+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    package="ml.dmlc.tvm.android.demo" >
+
+    <uses-permission android:name="android.permission.CAMERA" />
+    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
+    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
+
+    <application
+        android:allowBackup="true"
+        android:label="@string/app_name"
+        android:supportsRtl="true"
+        android:theme="@style/AppTheme" >
+        <activity
+            android:name=".MainActivity"
+            android:label="@string/app_name"
+            android:theme="@style/AppTheme.NoActionBar"
+            android:screenOrientation="portrait">
+            <intent-filter>
+                <action android:name="android.intent.action.MAIN" />
+                <category android:name="android.intent.category.LAUNCHER" />
+            </intent-filter>
+        </activity>
+        <provider
+            android:name="android.support.v4.content.FileProvider"
+            android:authorities="${applicationId}.provider"
+            android:exported="false"
+            android:grantUriPermissions="true">
+            <meta-data
+                android:name="android.support.FILE_PROVIDER_PATHS"
+                android:resource="@xml/provider_paths"/>
+        </provider>
+    </application>
+
+    <uses-permission android:name="android.permission.INTERNET" />
+
+</manifest>
\ No newline at end of file

apps/android_deploy/app/src/main/jni/Android.mk

+LOCAL_PATH := $(call my-dir)
+MY_PATH := $(LOCAL_PATH)
+
+include $(CLEAR_VARS)
+
+LOCAL_PATH := $(MY_PATH)
+ROOT_PATH := $(MY_PATH)/../../../../../..
+
+ifndef config
+	ifneq ("$(wildcard ./config.mk)","")
+	  config ?= config.mk
+	else
+	  config ?= make/config.mk
+	endif
+endif
+
+include $(config)
+
+LOCAL_SRC_FILES := ml_dmlc_tvm_native_c_api.cc
+LOCAL_LDFLAGS := -L$(SYSROOT)/usr/lib/ -llog
+
+LOCAL_C_INCLUDES := $(ROOT_PATH)/include \
+                    $(ROOT_PATH)/dlpack/include \
+                    $(ROOT_PATH)/dmlc-core/include \
+                    $(ROOT_PATH)/HalideIR/src \
+                    $(ROOT_PATH)/topi/include
+
+LOCAL_MODULE = tvm4j_runtime_packed
+
+LOCAL_CPP_FEATURES += exceptions
+LOCAL_LDLIBS += -latomic
+LOCAL_ARM_MODE := arm
+
+ifdef ADD_C_INCLUDES
+	LOCAL_C_INCLUDES += $(ADD_C_INCLUDES)
+endif
+
+ifdef ADD_LDLIBS
+	LOCAL_LDLIBS += $(ADD_LDLIBS)
+endif
+
+include $(BUILD_SHARED_LIBRARY)

apps/android_deploy/app/src/main/jni/Application.mk

+ifndef config
+	ifneq ("$(wildcard ./config.mk)","")
+	  config ?= config.mk
+	else
+	  config ?= make/config.mk
+	endif
+endif
+
+include $(config)
+
+APP_STL := gnustl_static
+
+APP_CPPFLAGS += -DDMLC_LOG_STACK_TRACE=0 -DTVM4J_ANDROID=1 -std=c++11 -Oz -frtti
+ifeq ($(USE_OPENCL), 1)                                                                                                                                             
+	APP_CPPFLAGS += -DTVM_OPENCL_RUNTIME=1
+endif

apps/android_deploy/app/src/main/jni/build.sh

+#!/bin/bash
+PATH="$PATH:/usr/local/bin"
+CURR_DIR=$(cd `dirname $0`; pwd)
+ROOT_DIR="$CURR_DIR/../../../../../.."
+javah -o $CURR_DIR/ml_dmlc_tvm_native_c_api.h -cp "$ROOT_DIR/jvm/core/target/*" ml.dmlc.tvm.LibInfo || exit -1
+cp -f $ROOT_DIR/jvm/native/src/main/native/ml_dmlc_tvm_native_c_api.cc $CURR_DIR/ || exit -1
+cp -f $ROOT_DIR/jvm/native/src/main/native/jni_helper_func.h $CURR_DIR/ || exit -1
+rm -rf $CURR_DIR/../libs
+ndk-build --directory=$CURR_DIR

apps/android_deploy/app/src/main/jni/make/config.mk

+#-------------------------------------------------------------------------------
+#  Template configuration for compiling
+#
+#  If you want to change the configuration, please use the following
+#  steps. Assume you are on the root directory. First copy the this
+#  file so that any local changes will be ignored by git
+#
+#  cp make/config.mk .
+#
+#  Next modify the according entries, and then compile by
+#
+#  ./build.sh
+#
+#-------------------------------------------------------------------------------
+APP_ABI = all
+
+APP_PLATFORM = android-17
+
+# whether enable OpenCL during compile
+USE_OPENCL = 0
+
+# the additional include headers you want to add, e.g., SDK_PATH/adrenosdk/Development/Inc
+ADD_C_INCLUDES =
+
+# the additional link libs you want to add, e.g., ANDROID_LIB_PATH/libOpenCL.so
+ADD_LDLIBS =

apps/android_deploy/app/src/main/jni/tvm_runtime.h

+/*!
+ *  Copyright (c) 2018 by Contributors
+ * \file tvm_runtime.h
+ * \brief Pack all tvm runtime source files
+ */
+#include <sys/stat.h>
+#include <fstream>
+
+#include "../src/runtime/c_runtime_api.cc"
+#include "../src/runtime/cpu_device_api.cc"
+#include "../src/runtime/workspace_pool.cc"
+#include "../src/runtime/module_util.cc"
+#include "../src/runtime/system_lib_module.cc"
+#include "../src/runtime/module.cc"
+#include "../src/runtime/registry.cc"
+#include "../src/runtime/file_util.cc"
+#include "../src/runtime/dso_module.cc"
+#include "../src/runtime/thread_pool.cc"
+#include "../src/runtime/threading_backend.cc"
+
+#include "../src/runtime/graph/graph_runtime.cc"
+
+#ifdef TVM_OPENCL_RUNTIME
+#include "../src/runtime/opencl/opencl_device_api.cc"
+#include "../src/runtime/opencl/opencl_module.cc"
+#endif

apps/android_deploy/app/src/main/res/layout/activity_main.xml

+<?xml version="1.0" encoding="utf-8"?>
+<android.support.design.widget.CoordinatorLayout
+    xmlns:android="http://schemas.android.com/apk/res/android"
+    xmlns:app="http://schemas.android.com/apk/res-auto"
+    xmlns:tools="http://schemas.android.com/tools"
+    android:layout_width="match_parent"
+    android:layout_height="match_parent"
+    tools:context="ml.dmlc.tvm.android.demo.MainActivity">
+
+    <android.support.design.widget.AppBarLayout
+        android:layout_height="wrap_content"
+        android:layout_width="match_parent"
+        android:theme="@style/AppTheme.AppBarOverlay">
+
+        <android.support.v7.widget.Toolbar
+            android:id="@+id/toolbar"
+            android:layout_width="match_parent"
+            android:layout_height="?attr/actionBarSize"
+            android:background="?attr/colorPrimary"
+            app:popupTheme="@style/AppTheme.PopupOverlay" />
+
+    </android.support.design.widget.AppBarLayout>
+
+    <include layout="@layout/content_main"/>
+
+</android.support.design.widget.CoordinatorLayout>
+

apps/android_deploy/app/src/main/res/layout/content_main.xml

+<?xml version="1.0" encoding="utf-8"?>
+<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
+              xmlns:tools="http://schemas.android.com/tools"
+              xmlns:app="http://schemas.android.com/apk/res-auto"
+              android:orientation="vertical"
+              android:layout_width="fill_parent"
+              android:layout_height="wrap_content"
+              app:layout_behavior="@string/appbar_scrolling_view_behavior"
+              tools:showIn="@layout/activity_main">
+
+    <LinearLayout
+        android:layout_width="match_parent"
+        android:layout_height="match_parent"
+        android:orientation="horizontal">
+
+        <Button
+            android:id="@+id/btnPickImage"
+            android:layout_width="match_parent"
+            android:layout_height="wrap_content"
+            android:layout_weight="1"
+            android:text="Select or Capture picture" />
+
+    </LinearLayout>
+    <View
+        android:layout_width="match_parent"
+        android:layout_height="4dp" />
+    <TextView
+        android:id="@+id/resultTextView"
+        android:layout_width="match_parent"
+        android:layout_height="60dp"
+        android:layout_weight="1"
+        android:textColor="@color/colorPrimary"
+        android:textSize="15sp" />
+    <View
+        android:layout_width="match_parent"
+        android:layout_height="4dp" />
+    <ImageView
+        android:id="@+id/imageView"
+        android:layout_width="match_parent"
+        android:layout_height="375dp"
+        android:layout_weight="1" />
+
+    <View
+        android:layout_width="match_parent"
+        android:layout_height="10dp" />
+</LinearLayout>

apps/android_deploy/app/src/main/res/values/colors.xml

+<?xml version="1.0" encoding="utf-8"?>
+<resources>
+    <color name="colorPrimary">#3F51B5</color>
+    <color name="colorPrimaryDark">#303F9F</color>
+    <color name="colorAccent">#06d467</color>
+</resources>

apps/android_deploy/app/src/main/res/values/strings.xml

+<resources>
+    <string name="app_name">TVM Android Demo</string>
+</resources>
\ No newline at end of file

apps/android_deploy/app/src/main/res/values/styles.xml

+<resources>
+
+    <!-- Base application theme. -->
+    <style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
+        <!-- Customize your theme here. -->
+        <item name="colorPrimary">@color/colorPrimary</item>
+        <item name="colorPrimaryDark">@color/colorPrimaryDark</item>
+        <item name="colorAccent">@color/colorAccent</item>
+    </style>
+    <style name="AppTheme.NoActionBar">
+        <item name="windowActionBar">false</item>
+        <item name="windowNoTitle">true</item>
+    </style>
+    <style name="AppTheme.AppBarOverlay" parent="ThemeOverlay.AppCompat.Dark.ActionBar" />
+    <style name="AppTheme.PopupOverlay" parent="ThemeOverlay.AppCompat.Light" />
+
+</resources>

apps/android_deploy/app/src/main/res/xml/provider_paths.xml

+<?xml version="1.0" encoding="utf-8"?>
+<paths xmlns:android="http://schemas.android.com/apk/res/android">
+    <external-path name="external_files" path="."/>
+</paths>
\ No newline at end of file

apps/android_deploy/build.gradle

+// Top-level build file where you can add configuration options common to all sub-projects/modules.
+
+buildscript {
+    repositories {
+        jcenter()
+    }
+    dependencies {
+        classpath 'com.android.tools.build:gradle:2.3.3'
+        classpath 'org.apache.httpcomponents:httpclient:4.5.4'
+
+        // NOTE: Do not place your application dependencies here; they belong
+        // in the individual module build.gradle files
+    }
+}
+
+allprojects {
+    repositories {
+        jcenter()
+        maven {
+          url 'https://maven.google.com'
+        }
+        mavenLocal()
+        mavenCentral()
+    }
+}
+
+task clean(type: Delete) {
+    delete rootProject.buildDir
+}

apps/android_deploy/dev_tools/gen_keystore.sh

+#!/bin/bash
+CURR_DIR=$(cd `dirname $0`; pwd)
+keytool -genkey -keystore $CURR_DIR/tvmdemo.keystore -alias tvmdemo -keyalg RSA -validity 10000

apps/android_deploy/dev_tools/sign_apk.sh

+#!/bin/bash
+CURR_DIR=$(cd `dirname $0`; pwd)
+APK_DIR=$CURR_DIR/../app/build/outputs/apk
+UNSIGNED_APK=$APK_DIR/app-release-unsigned.apk
+SIGNED_APK=$APK_DIR/tvmdemo-release.apk
+jarsigner -verbose -keystore $CURR_DIR/tvmdemo.keystore -signedjar $SIGNED_APK $UNSIGNED_APK 'tvmdemo'
+echo $SIGNED_APK

apps/android_deploy/settings.gradle

+include ':app'

docs/how_to/deploy_android.md

+# How to deploy and use compiled model on Android
+
+
+This tutorial explain below aspects (Unlike the android_rpc approach we already have)
+
+  * Build a model for android target
+  * TVM run on Android using Java API
+
+As an example here is a reference block diagram.
+
+![](http://www.tvmlang.org/images/release/tvm_flexible.png)
+
+## Build model for Android Target
+
+NNVM compilation of model for android target could follow same approach like android_rpc.
+
+An reference exampe can be found at [chainer-nnvm-example](https://github.com/tkat0/chainer-nnvm-example)
+
+Above example will directly run the compiled model on RPC target. Below modification at [rum_mobile.py](https://github.com/tkat0/chainer-nnvm-example/blob/5b97fd4d41aa4dde4b0aceb0be311054fb5de451/run_mobile.py#L64) will save the compilation output which is required on android target.
+
+```
+lib.export_library("deploy_lib.so", ndk.create_shared)
+with open("deploy_graph.json", "w") as fo:
+    fo.write(graph.json())
+with open("deploy_param.params", "wb") as fo:
+    fo.write(nnvm.compiler.save_param_dict(params))
+```
+
+deploy_lib.so, deploy_graph.json, deploy_param.params will go to android target.
+
+## TVM run on Android using Java API
+### TVM Runtime for android Target
+
+Refer [here](https://github.com/dmlc/tvm/blob/master/apps/android_deploy/README.md#build-and-installation) to build CPU/OpenCL version flavor TVM runtime for android target.
+
+### Android Native API Reference
+
+From android java TVM API to load model & execute can be refered at this [java](https://github.com/dmlc/tvm/blob/master/apps/android_deploy/app/src/main/java/ml/dmlc/tvm/android/demo/MainActivity.java) sample source.
+
+