How to submit a python batch job?

[HaoYang670]

cc @pan3793 Thank you for telling me that Kyuubi batch API supports python, which helps a lot for our project!

I also got this error in log when submit a Python job using the post batch api. The python file is located at the Kyuubi server.

org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:97)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 19 more

I've tried to set the batch type to be spark and pyspark but neither of them works.
Also, how to set the className?

[joaopamaral]

Hi @HaoYang670, I was able to run a python code with the POST /batches (with uploading resource) (to upload the python script) running the following code:

from requests_toolbelt.multipart.encoder import MultipartEncoder
import json

encoder = MultipartEncoder(
    fields={
    'batchRequest': ('', json.dumps(
        {
            'batchType': 'pyspark',
            'name': 'my_python_script_app',
            'conf': {
                'spark.submit.deployMode': 'cluster'
            }
        }
    ), 'application/json'),
    'resourceFile': ('script_file.py', open('/tmp/script_file.py', 'rb')),
})


requests.post(f'{KYUUBI_URL}/api/v1/batches', 
              data=encoder,
              headers={'Content-Type': encoder.content_type})

Unfortunately, I didn't try with a file located on the server.