cos ranger version

tencentyun · Dec 15, 2020 · 8284af2 · 8284af2
1 parent 9ee8abc
commit 8284af2
Show file tree

Hide file tree

Showing 29 changed files with 763 additions and 122 deletions.
diff --git a/README.md b/README.md
@@ -198,8 +198,8 @@ done
 
 | 属性键                             | 说明                | 默认值 | 必填项 |
 |:-----------------------------------:|:--------------------|:-----:|:---:|
-|fs.defaultFS                       | 配置hadoop默认使用的底层文件系统，如果想使用cos作为hadoop默认文件系统，则此项应设置为cosn://bucket-appid，此时可以通过文件路径访问cos对象，如/hadoop/inputdata/test.dat。若不想把cos作为hadoop默认文件系统，则不需要修改此项，当需要访问cos上的对象时，则指定完整的uri即可，如cosn://testbucket-1252681927/hadoop/inputdata/test.dat来访问。
-|fs.cosn.credentials.provider       |配置secret id和secret key的获取方式。当前支持三种获取方式：1.org.apache.hadoop.fs.auth.SessionCredentialProvider：从请求URI中获取secret id和secret key，其格式为：cosn://{secretId}:{secretKey}@examplebucket-1250000000000/; 2.org.apache.hadoop.fs.auth.SimpleCredentialProvider：从core-site.xml配置文件中读取fs.cosn.userinfo.secretId和fs.cosn.userinfo.secretKey来获取secret id和secret key; 3.org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider：从系统环境变量COS_SECRET_ID和COS_SECRET_KEY中获取；4.org.apache.hadoop.fs.auth.CVMInstanceCredentialsProvider：利用腾讯云云服务器（CVM）绑定的角色，获取访问COS的临时密钥; 5. org.apache.hadoop.fs.auth.CPMInstanceCredentialsProvider：利用腾讯云黑石物理机（CPM）绑定的角色，获取访问COS的临时密钥。|如果不指定改配置项，默认会按照以下顺序读取：1.org.apache.hadoop.fs.auth.SessionCredentialProvider; 2.org.apache.hadoop.fs.auth.SimpleCredentialProvider；3.org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider; 4.org.apache.hadoop.fs.auth.CVMInstanceCredentialsProvider; 5.org.apache.hadoop.fs.auth.CPMInstanceCredentialsProvider|否|
+|fs.defaultFS                      | 配置hadoop默认使用的底层文件系统，如果想使用cos作为hadoop默认文件系统，则此项应设置为cosn://bucket-appid，此时可以通过文件路径访问cos对象，如/hadoop/inputdata/test.dat。若不想把cos作为hadoop默认文件系统，则不需要修改此项，当需要访问cos上的对象时，则指定完整的uri即可，如cosn://testbucket-1252681927/hadoop/inputdata/test.dat来访问。|
+|fs.cosn.credentials.provider       |配置secret id和secret key的获取方式。当前支持三种获取方式：1.org.apache.hadoop.fs.auth.SessionCredentialProvider：从请求URI中获取secret id和secret key，其格式为：cosn://{secretId}:{secretKey}@examplebucket-1250000000000/; 2.org.apache.hadoop.fs.auth.SimpleCredentialProvider：从core-site.xml配置文件中读取fs.cosn.userinfo.secretId和fs.cosn.userinfo.secretKey来获取secret id和secret key; 3.org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider：从系统环境变量COS_SECRET_ID和COS_SECRET_KEY中获取；4.org.apache.hadoop.fs.auth.SessionTokenCredentialProvider: 设置token；5.org.apache.hadoop.fs.auth.CVMInstanceCredentialsProvider：利用腾讯云云服务器（CVM）绑定的角色，获取访问COS的临时密钥; 6. org.apache.hadoop.fs.auth.CPMInstanceCredentialsProvider：利用腾讯云黑石物理机（CPM）绑定的角色，获取访问COS的临时密钥。7. org.apache.hadoop.fs.auth.RangerCredentialsProvider 使用ranger进行获取秘钥 |如果不指定改配置项，默认会按照以下顺序读取：1.org.apache.hadoop.fs.auth.SessionCredentialProvider; 2.org.apache.hadoop.fs.auth.SimpleCredentialProvider;3.org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider;4.org.apache.hadoop.fs.auth.SessionTokenCredentialProvider；5.org.apache.hadoop.fs.auth.CVMInstanceCredentialsProvider;6.org.apache.hadoop.fs.auth.CPMInstanceCredentialsProvider|否|
 |fs.cosn.useHttps|配置是否使用https协议。|false|否|
 |fs.cosn.bucket.endpoint_suffix|指定要连接的COS endpoint，该项为非必填项目。对于公有云COS用户而言，只需要正确填写上述的region配置即可。兼容原配置项：fs.cosn.userinfo.endpoint_suffix。|无|否|
 |fs.cosn.userinfo.secretId/secretKey| 填写您账户的API 密钥信息。可通过 [云 API 密钥 控制台](https://console.cloud.tencent.com/capi) 查看。| 无  | 是|

diff --git a/compile.sh b/compile.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+base_dir=$(cd `dirname $0`;pwd)
+cd ${base_dir}
+hadoop_version_array=("2.6.5" "2.7.5" "2.8.5" "3.1.0" "3.3.0")
+
+origin_version=$(mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive exec:exec)
+
+for hadoop_version in ${hadoop_version_array[@]}
+do
+    sed -i -E "s/<hadoop\.version>.*<\/hadoop\.version>/<hadoop\.version>${hadoop_version}<\/hadoop\.version>/g" pom.xml
+    mvn versions:set -DnewVersion=${hadoop_version}-${origin_version}
+    mvn clean verify
+    rm -rf dep/${hadoop_version}
+    mkdir -p dep/${hadoop_version}
+    cp target/*.jar dep/${hadoop_version}/
+    cp target/*.asc dep/${hadoop_version}/
+    cp target/*.pom dep/${hadoop_version}/
+
+    mvn versions:set -DnewVersion=${origin_version}
+done
diff --git a/pom.xml b/pom.xml
@@ -6,7 +6,7 @@
 
     <groupId>com.qcloud.cos</groupId>
     <artifactId>hadoop-cos</artifactId>
-    <version>3.1.0-5.8.7</version>
+    <version>5.9.0</version>
     <packaging>jar</packaging>
 
     <name>Apache Hadoop Tencent Qcloud COS Support</name>
@@ -38,9 +38,9 @@
 
     <properties>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <maven.compiler.source>1.8</maven.compiler.source>
-        <maven.compiler.target>1.8</maven.compiler.target>
-        <hadoop.version>3.1.0</hadoop.version>
+        <maven.compiler.source>1.7</maven.compiler.source>
+        <maven.compiler.target>1.7</maven.compiler.target>
+        <hadoop.version>3.3.0</hadoop.version>
         <cos_api.version>5.6.32</cos_api.version>
         <google.guava.version>24.1.1-jre</google.guava.version>
         <commons_lang3.version>3.1</commons_lang3.version>
@@ -80,10 +80,31 @@
 
     <build>
         <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-source-plugin</artifactId>
+                <version>3.2.0</version>
+                <configuration>
+                    <attach>true</attach>
+                </configuration>
+                <executions>
+                    <execution>
+                        <phase>compile</phase>
+                        <goals>
+                            <goal>jar</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+
+
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-javadoc-plugin</artifactId>
                 <version>2.9.1</version>
+                <configuration>
+                    <additionalJOption>-Xdoclint:none</additionalJOption>
+                </configuration>
                 <executions>
                     <execution>
                         <id>javadoc-jar</id>
@@ -126,5 +147,4 @@
             </plugin>
         </plugins>
     </build>
-
 </project>
diff --git a/src/main/java/org/apache/hadoop/fs/BufferPool.java b/src/main/java/org/apache/hadoop/fs/BufferPool.java
@@ -258,6 +258,9 @@ public void returnBuffer(CosNByteBuffer buffer)
         }
     }
 
+    /**
+     *  close
+     */
     public synchronized void close() {
         LOG.info("Close a buffer pool instance.");
 

diff --git a/src/main/java/org/apache/hadoop/fs/CosEncryptionMethods.java b/src/main/java/org/apache/hadoop/fs/CosEncryptionMethods.java
@@ -25,22 +25,14 @@ public String getMethod() {
         return method;
     }
 
-    /**
-     * Flag to indicate this is a server-side encryption option.
-     * @return true if this is server side.
-     */
-    public boolean isServerSide() {
-        return serverSide;
-    }
-
     /**
      * Get the encryption mechanism from the value provided.
      * @param name algorithm name
      * @return the method
      * @throws IOException if the algorithm is unknown
      */
     public static CosEncryptionMethods getMethod(String name) throws IOException {
-        if(StringUtils.isNullOrEmpty(name)) {
+        if (StringUtils.isNullOrEmpty(name)) {
             return NONE;
         }
         for (CosEncryptionMethods v : values()) {
@@ -50,4 +42,13 @@ public static CosEncryptionMethods getMethod(String name) throws IOException {
         }
         throw new IOException(UNKNOWN_ALGORITHM_MESSAGE + name);
     }
+
+    /**
+     * Flag to indicate this is a server-side encryption option.
+     * @return true if this is server side.
+     */
+    public boolean isServerSide() {
+        return serverSide;
+    }
+
 }
-Original file line number
+Diff line change
@@ Expand Up / @@ -258,6 +258,9 @@ public void returnBuffer(CosNByteBuffer buffer) @@
             }
         }
+        /**
+         *  close
+         */
         public synchronized void close() {
             LOG.info("Close a buffer pool instance.");
@@ Expand Down @@