Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error streaming to GBQ #168

Open
aplsms opened this issue Oct 3, 2018 · 1 comment
Open

Error streaming to GBQ #168

aplsms opened this issue Oct 3, 2018 · 1 comment

Comments

@aplsms
Copy link

aplsms commented Oct 3, 2018

Hello,

I'm doing migration form fluentd 0.12 to 1.0

I have error message like " message="Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the error stream for more details." reason="invalid"

How to get error stream for details?

Error message in fluentd logs:

2018-10-03 05:11:16 +0000 [error]: #0 job.insert API (rows) job_id="job_0pb2mErCjSc6aXMxEXMiIaWkE1gF" project_id="voltaic-phalanx-757" dataset="prod_logs" table="analyst_server_20181003" message="Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the error stream for more details." reason="invalid"
2018-10-03 05:11:16 +0000 [error]: #0 job.insert API (rows) job_id="job_0pb2mErCjSc6aXMxEXMiIaWkE1gF" project_id="voltaic-phalanx-757" dataset="prod_logs" table="analyst_server_20181003" message="Error while reading data, error message: JSON parsing error in row starting at position 0: No such field: stack_trace." reason="invalid"
2018-10-03 05:11:16 +0000 [error]: #0 job.insert API (result) job_id="job_0pb2mErCjSc6aXMxEXMiIaWkE1gF" project_id="voltaic-phalanx-757" dataset="prod_logs" table="analyst_server_20181003" message="Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the error stream for more details." reason="invalid"
2018-10-03 05:11:16 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=0 records=10 error_class=Fluent::BigQuery::UnRetryableError error="failed to load into bigquery, and cannot retry"
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/bigquery/writer.rb:223:in `wait_load_job'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/bigquery/writer.rb:170:in `create_load_job'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:453:in `block in load'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:485:in `block in create_upload_source'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:484:in `open'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:484:in `create_upload_source'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:452:in `load'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:446:in `_write'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:340:in `write'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1099:in `try_flush'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1378:in `flush_thread_run'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:440:in `block (2 levels) in start'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Environments

  • fluentd version: td-agent 3.2.0-0
  • plugin version: fluent-plugin-bigquery (1.2.0)

Configuration

   <store>
     @type bigquery
     method load

     <buffer time>
       @type file
       path /fluentd/buffers/GBQ/bigquery.*.buffer
       timekey           24h
       timekey_wait      10m
       flush_interval 30m
       flush_at_shutdown true
       timekey_use_utc
     </buffer>
     <inject>
       time_key time
       time_type string
       # time_format %Y-%m-%d %H:%M:%S
       time_format %s
     </inject>

     auth_method json_key
     json_key json_key_path.json

     auth_method private_key
     email [email protected]
     private_key_path /fluentd/config/prod.ta-google.p12

     project voltaic-phalanx-757
     dataset prod_logs
     table analyst_server_%Y%m%d
     auto_create_table true
     schema [
       {"name": "hostname", "type": "STRING"},
       {"name": "decisionId", "type": "STRING"},
       {"name": "debtorId", "type": "STRING"},
       {"name": "message", "type": "STRING"},
       {"name": "time", "type": "TIMESTAMP"}
     ]
   </store>

old config file that working well:

  <store>
    type bigquery
    project voltaic-phalanx-757
    dataset prod_logs
    table analyst_server_%Y%m%d
    auto_create_table true

    time_format %s
    time_field  time
    field_timestamp time
    field_string hostname,decisionId,debtorId,message

    auth_method private_key
    email [email protected]
    private_key_path /ebs/config/prod.ta-google.p12
  </store>

Expected Behavior

json data appeared in the GBQ

Actual Behavior

Error message="Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the error stream for more details." reason="invalid"

Log (if you have)

2018-10-03 05:11:16 +0000 [error]: #0 job.insert API (rows) job_id="job_0pb2mErCjSc6aXMxEXMiIaWkE1gF" project_id="voltaic-phalanx-757" dataset="prod_logs" table="analyst_server_20181003" message="Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the error stream for more details." reason="invalid"
2018-10-03 05:11:16 +0000 [error]: #0 job.insert API (rows) job_id="job_0pb2mErCjSc6aXMxEXMiIaWkE1gF" project_id="voltaic-phalanx-757" dataset="prod_logs" table="analyst_server_20181003" message="Error while reading data, error message: JSON parsing error in row starting at position 0: No such field: stack_trace." reason="invalid"
2018-10-03 05:11:16 +0000 [error]: #0 job.insert API (result) job_id="job_0pb2mErCjSc6aXMxEXMiIaWkE1gF" project_id="voltaic-phalanx-757" dataset="prod_logs" table="analyst_server_20181003" message="Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the error stream for more details." reason="invalid"
2018-10-03 05:11:16 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=0 records=10 error_class=Fluent::BigQuery::UnRetryableError error="failed to load into bigquery, and cannot retry"
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/bigquery/writer.rb:223:in `wait_load_job'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/bigquery/writer.rb:170:in `create_load_job'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:453:in `block in load'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:485:in `block in create_upload_source'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:484:in `open'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:484:in `create_upload_source'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:452:in `load'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:446:in `_write'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-bigquery-1.2.0/lib/fluent/plugin/out_bigquery.rb:340:in `write'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1099:in `try_flush'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1378:in `flush_thread_run'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:440:in `block (2 levels) in start'
  2018-10-03 05:11:16 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
@aplsms
Copy link
Author

aplsms commented Oct 3, 2018

just note: in original log we have much more fields, but in the table we have fields exactly listed in schema.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aplsms