Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hashcheck=true not embedding the hashes inside objects for s3. #406

Open
karasjoh000 opened this issue Feb 10, 2021 · 0 comments
Open

hashcheck=true not embedding the hashes inside objects for s3. #406

karasjoh000 opened this issue Feb 10, 2021 · 0 comments

Comments

@karasjoh000
Copy link

karasjoh000 commented Feb 10, 2021

Hashchecks are enabled

<!-- following the example from https://github.com/intel-cloud/cosbench/blob/master/release/conf/hashcheck.xml -->
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="19-1GB" description="sample benchmark for s3">
  <storage type="s3" config="accesskey=<>;secretkey=<>;endpoint=http://<>;path_style_access=true"/>
  <workflow>
    <workstage name="cleanup">
      <work type="cleanup" workers="30" config="cprefix=ssds3;containers=r(1,2);objects=r(1,8000)" />
    </workstage>
    <workstage name="dispose">
      <work type="dispose" workers="1" config="cprefix=ssds3;containers=r(1,2)" />
    </workstage>
    <workstage name="init">
      <work type="init" workers="1" config="cprefix=ssds3;containers=r(1,2)" />
    </workstage>
    <!-- <workstage name="prepare" division="object">
      <work type="prepare" workers="30" config="hashcheck=true;cprefix=ssds3;containers=r(1,2);objects=r(1,1000);sizes=c(10)MB" />
    </workstage> -->
    <workstage name="prepare" division="object">
      <work name="main" workers="19" totalBytes="200000000000" division="object">
        <operation type="write" ratio="100" config="hashcheck=true;cprefix=ssds3;containers=s(1,2);objects=s(1,40);sizes=c(1)GB" />
      </work>
    </workstage>

    <workstage name="main">
      <work name="main" workers="19" totalBytes="400000000000" division="object">
        <operation type="write" ratio="50" config="hashcheck=true;cprefix=ssds3;containers=s(1,2);objects=s(41,80);sizes=c(1)GB" />
        <operation type="read" ratio="50" config="hashcheck=true;cprefix=ssds3;containers=s(1,2);objects=s(1,40)"/>
      </work>
    </workstage>
  </workflow>
</workload>

But the objects have no hash stored at the tail !!!!<hash>!!!!

awse s3 cp s3://ssds31/myobjects1 check1
# download: s3://ssds31/myobjects1 to ./check1
tail -c 100 check1
# vgbcwguswthozwkuojbqfozebtgulkcrtbuqthgkveeaqwvtmfmrvjwplhnebpbhjpblxvtqerwpliodlguxepaamhwgmoiahfdy

This is for ceph rgw. Note I did apply this suggestion #320 by adding
-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true
otherwise, the parallel reads and writes did not work at all.

karasjoh000 added a commit to karasjoh000/zenko-cosbench that referenced this issue Feb 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant