Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exporter service failing with "slice index out of range" error #42

Open
davidgreen4 opened this issue Nov 28, 2019 · 4 comments
Open

Exporter service failing with "slice index out of range" error #42

davidgreen4 opened this issue Nov 28, 2019 · 4 comments

Comments

@davidgreen4
Copy link

Hi

The oracle prometheus exporter service is failing after a few days on several of our servers. The error in the messages file is below...

Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: panic: reflect: slice index out of range [recovered]
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: panic: reflect: slice index out of range
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: goroutine 560646 [running]:
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.handleErr(0xc00085db98)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/yaml.go:249 +0x9a
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: panic(0x8dd840, 0xa3f140)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /usr/lib/go-1.12/src/runtime/panic.go:522 +0x1b5
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: reflect.Value.Index(0x8d3780, 0x10062a0, 0x197, 0xe, 0xc0000d5408, 0x199, 0x199)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /usr/lib/go-1.12/src/reflect/value.go:925 +0x20c
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).slicev(0xc0000d8000, 0x0, 0x0, 0x8d3780, 0x10062a0, 0x197)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:272 +0x1c1
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).marshal(0xc0000d8000, 0x0, 0x0, 0x8d3780, 0x10062a0, 0x197)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:166 +0x404
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).structv.func1()
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:226 +0x299
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).mappingv(0xc0000d8000, 0x0, 0x0, 0xc0000e5530)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:256 +0x147
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).structv(0xc0000d8000, 0x0, 0x0, 0x91e480, 0x10062a0, 0x199)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:213 +0xf7
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).marshal(0xc0000d8000, 0x0, 0x0, 0x91e480, 0x10062a0, 0x199)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:160 +0x718
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).marshal(0xc0000d8000, 0x0, 0x0, 0x8c75e0, 0x10062a0, 0x16)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:154 +0x591
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.(*encoder).marshalDoc(0xc0000d8000, 0x0, 0x0, 0x8c75e0, 0x10062a0, 0x16)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/encode.go:93 +0x10f
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: gopkg.in/yaml%2ev2.Marshal(0x8c75e0, 0x10062a0, 0x0, 0x0, 0x0, 0x0, 0x0)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/gopkg.in/yaml.v2/yaml.go:203 +0x329
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: main.WriteAccess()
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/github.com/freenetgigital/prometheus_oracle_exporter/misc.go:91 +0x3d
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: main.(*Exporter).ScrapeAlertlog(0xc000150300)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/github.com/freenetgigital/prometheus_oracle_exporter/alertlog.go:152 +0xba6
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: main.(*Exporter).Collect(0xc000150300, 0xc0004ad320)
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/github.com/freenetgigital/prometheus_oracle_exporter/main.go:881 +0x2d3
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/github.com/prometheus/client_golang/prometheus/registry.go:442 +0x19d
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
Nov 25 15:48:54 cs-tc3-oracle-liv1 prometheus_oracle_exporter-amd64[4005]: /home/mneumann3/go/src/github.com/prometheus/client_golang/prometheus/registry.go:453 +0x56e
Nov 25 15:48:54 cs-tc3-oracle-liv1 systemd[1]: systemd-oracledp-exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 25 15:49:01 cs-tc3-oracle-liv1 systemd[1]: Created slice User Slice of root.

the service file looks like this...

[Unit]
Description=Oracle Database Exporter for Prometheus
DefaultDependencies=no

[Service]
User=oracle
Group=oinstall
Environment=ORACLE_HOME=/u01/app/oracle/product/18.4.0/dbhome_1
Environment=LD_LIBRARY_PATH=/u01/app/oracle/product/18.4.0/dbhome_1/lib
Environment=PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/oracle/Scripts:/u01/app/oracle/product/18.4.0/dbhome_1/bin
Type=simple
RemainAfterExit=yes
ExecStart=/home/oracle/prometheus_oracle_exporter/prometheus_oracle_exporter-amd64 -configfile=/home/oracle/prometheus_oracle_exporter/oracle.conf -web.listen-address :9161

any ideas what the problem would be ??
@floyd871
Copy link
Owner

Seems like a problem with scraping the alertlog. So far i never used the exporter for Oracle 18/19. I've only used it in 11g/12g. Can you please post your configfile and a snippet of the alertlog?

@davidgreen4
Copy link
Author

hi, the alert log seems to be swamped with TNS messages, i'm going to investigate those. The config file is monitoring 20 databases so it's difficult to pin down if one of the messages is the alert logs corresponds to the error message in the messages file. I'm going to limit the config to just monitor a couple of databases and see if that make things easier to piece together what could be causing the problem.

will post again when things are a bit simpler

thanks for you help

@davidgreen4
Copy link
Author

davidgreen4 commented Dec 4, 2019 •
edited
Loading

Hi, I've cleaned up the TNS messages from the alerts logs, and also excluded the particular error (TNS-12535) from being recorded by the exporter. The configfile is attached, from that you can see it is monitoring 3 databases, there is nothing in the alert log anywhere near the time of the failure in the exporter service.

this is the message displayed by the systemctl status for the service

Dec 02 11:04:55 cs-tc3-oracle-dev2.apak-dmz.local systemd[1]: Started Oracle Database Exporter for Prometheus.
Dec 02 11:04:55 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: time="2019-12-02T11:04:55Z" level=info msg="Starting Prometheus Oracle exporter 1.1.3" source="main.go:944"
Dec 02 11:04:55 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: time="2019-12-02T11:04:55Z" level=info msg="Config loaded: /home/oracle/prometheus_oracle_exporter/oracle.conf" source="main.go:946"
Dec 02 11:04:55 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: time="2019-12-02T11:04:55Z" level=info msg="Listening on :9161" source="main.go:954"
Dec 04 03:03:32 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 04 03:03:32 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5f9297]
Dec 04 03:03:32 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: goroutine 965731 [running]:
Dec 04 03:03:32 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: database/sql.(*DB).Close(0x0, 0xc0000b5d78, 0x2)
Dec 04 03:03:32 cs-tc3-oracle-dev2.apak-dmz.local prometheus_oracle_exporter-amd64[31663]: /usr/lib/go-1.12/src/database/sql/sql.go:758 +0x37
Dec 04 03:03:32 cs-tc3-oracle-dev2.apak-dmz.local systemd[1]: systemd-oracledp-exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT

@davidgreen4
Copy link
Author

connections:

  • connection: ro_user/ro_user@localhost:1521/dev2dv30.cs-tc3-oracle-dev2
    database: dev2dv30
    instance: dev2dv30
    alertlog:

    • file: /u01/app/oracle/diag/rdbms/dev2dv30/dev2dv30/trace/alert_dev2dv30.log
      ignoreora:
      • ORA-01041
      • ORA-01555
      • ORA-28
      • ORA-235
      • ORA-609
      • ORA-3136
      • TNS-12535

  • connection: ro_user/ro_user@localhost:1521/dev2dv31.cs-tc3-oracle-dev2
    database: dev2dv31
    instance: dev2dv31
    alertlog:

    • file: /u01/app/oracle/diag/rdbms/dev2dv31/dev2dv31/trace/alert_dev2dv31.log
      ignoreora:
      • ORA-01041
      • ORA-01555
      • ORA-28
      • ORA-235
      • ORA-609
      • ORA-3136
      • TNS-12535

  • connection: ro_user/ro_user@localhost:1521/dev2dv34.cs-tc3-oracle-dev2
    database: dev2dv34
    instance: dev2dv34
    alertlog:

    • file: /u01/app/oracle/diag/rdbms/dev2dv34/dev2dv34/trace/alert_dev2dv34.log
      ignoreora:
      • ORA-01041
      • ORA-01555
      • ORA-28
      • ORA-235
      • ORA-609
      • ORA-3136
      • TNS-12535

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@floyd871 @davidgreen4