diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..1bc915c --- /dev/null +++ b/.gitignore @@ -0,0 +1,156 @@ +## Ignore Visual Studio temporary files, build results, and +## files generated by popular Visual Studio add-ons. + +# User-specific files +*.suo +*.user +*.sln.docstates + +# Build results + +[Dd]ebug/ +[Rr]elease/ +x64/ +build/ +[Bb]in/ +[Oo]bj/ + +# Enable "build/" folder in the NuGet Packages folder since NuGet packages use it for MSBuild targets +!packages/*/build/ + +# MSTest test Results +[Tt]est[Rr]esult*/ +[Bb]uild[Ll]og.* + +*_i.c +*_p.c +*.ilk +*.meta +*.obj +*.pch +*.pdb +*.pgc +*.pgd +*.rsp +*.sbr +*.tlb +*.tli +*.tlh +*.tmp +*.tmp_proj +*.log +*.vspscc +*.vssscc +.builds +*.pidb +*.log +*.scc + +# Visual C++ cache files +ipch/ +*.aps +*.ncb +*.opensdf +*.sdf +*.cachefile + +# Visual Studio profiler +*.psess +*.vsp +*.vspx + +# Guidance Automation Toolkit +*.gpState + +# ReSharper is a .NET coding add-in +_ReSharper*/ +*.[Rr]e[Ss]harper + +# TeamCity is a build add-in +_TeamCity* + +# DotCover is a Code Coverage Tool +*.dotCover + +# NCrunch +*.ncrunch* +.*crunch*.local.xml + +# Installshield output folder +[Ee]xpress/ + +# DocProject is a documentation generator add-in +DocProject/buildhelp/ +DocProject/Help/*.HxT +DocProject/Help/*.HxC +DocProject/Help/*.hhc +DocProject/Help/*.hhk +DocProject/Help/*.hhp +DocProject/Help/Html2 +DocProject/Help/html + +# Click-Once directory +publish/ + +# Publish Web Output +*.Publish.xml + +# NuGet Packages Directory +## TODO: If you have NuGet Package Restore enabled, uncomment the next line +#packages/ + +# Windows Azure Build Output +csx +*.build.csdef + +# Windows Store app package directory +AppPackages/ + +# Others +sql/ +*.Cache +ClientBin/ +[Ss]tyle[Cc]op.* +~$* +*~ +*.dbmdl +*.[Pp]ublish.xml +*.pfx +*.publishsettings + +# RIA/Silverlight projects +Generated_Code/ + +# Backup & report files from converting an old project file to a newer +# Visual Studio version. Backup files are not needed, because we have git ;-) +_UpgradeReport_Files/ +Backup*/ +UpgradeLog*.XML +UpgradeLog*.htm + +# SQL Server files +App_Data/*.mdf +App_Data/*.ldf + + +#LightSwitch generated files +GeneratedArtifacts/ +_Pvt_Extensions/ +ModelManifest.xml + +# ========================= +# Windows detritus +# ========================= + +# Windows image file caches +Thumbs.db +ehthumbs.db + +# Folder config file +Desktop.ini + +# Recycle Bin used on file shares +$RECYCLE.BIN/ + +# Mac desktop service store files +.DS_Store diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..cd38f32 --- /dev/null +++ b/README.md @@ -0,0 +1,5 @@ +#CollectdWin + +CollectdWin is a MS Windows service which collects, aggregates and publishes both application and system metrics periodically. These application and system metrics can be used for performance analysis and capacity planning. CollectdWin is very similar to Collectd (https://collectd.org), it was developed because Collectd doesnot support Windows. + +For more info, refer [CollectdWin docs](../../wiki) diff --git a/src/CollectdWinService.sln b/src/CollectdWinService.sln new file mode 100644 index 0000000..6ef06c5 --- /dev/null +++ b/src/CollectdWinService.sln @@ -0,0 +1,62 @@ + +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio 2013 +VisualStudioVersion = 12.0.21005.1 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "CollectdWinService", "CollectdWinService\CollectdWinService.csproj", "{D4244E6B-84EC-41A0-96A7-4F489F6098BC}" +EndProject +Project("{930C7802-8A8C-48F9-8165-68863BCCD9DD}") = "CollectdWin", "installer\CollectdWin.wixproj", "{93CC9E80-B6F5-4768-998E-889BDF5AB579}" + ProjectSection(ProjectDependencies) = postProject + {D4244E6B-84EC-41A0-96A7-4F489F6098BC} = {D4244E6B-84EC-41A0-96A7-4F489F6098BC} + EndProjectSection +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Debug|Mixed Platforms = Debug|Mixed Platforms + Debug|x64 = Debug|x64 + Debug|x86 = Debug|x86 + Release|Any CPU = Release|Any CPU + Release|Mixed Platforms = Release|Mixed Platforms + Release|x64 = Release|x64 + Release|x86 = Release|x86 + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|Any CPU.Build.0 = Debug|Any CPU + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|Mixed Platforms.ActiveCfg = Debug|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|Mixed Platforms.Build.0 = Debug|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|Mixed Platforms.Deploy.0 = Debug|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|x64.ActiveCfg = Debug|x64 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|x64.Build.0 = Debug|x64 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|x64.Deploy.0 = Debug|x64 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|x86.ActiveCfg = Debug|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|x86.Build.0 = Debug|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Debug|x86.Deploy.0 = Debug|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|Any CPU.ActiveCfg = Release|Any CPU + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|Any CPU.Build.0 = Release|Any CPU + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|Mixed Platforms.ActiveCfg = Release|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|Mixed Platforms.Build.0 = Release|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|x64.ActiveCfg = Release|x64 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|x64.Build.0 = Release|x64 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|x86.ActiveCfg = Release|x86 + {D4244E6B-84EC-41A0-96A7-4F489F6098BC}.Release|x86.Build.0 = Release|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Debug|Any CPU.ActiveCfg = Debug|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Debug|Mixed Platforms.ActiveCfg = Debug|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Debug|Mixed Platforms.Build.0 = Debug|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Debug|x64.ActiveCfg = Debug|x64 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Debug|x64.Build.0 = Debug|x64 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Debug|x86.ActiveCfg = Debug|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Debug|x86.Build.0 = Debug|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Release|Any CPU.ActiveCfg = Release|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Release|Mixed Platforms.ActiveCfg = Release|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Release|Mixed Platforms.Build.0 = Release|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Release|x64.ActiveCfg = Release|x64 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Release|x64.Build.0 = Release|x64 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Release|x86.ActiveCfg = Release|x86 + {93CC9E80-B6F5-4768-998E-889BDF5AB579}.Release|x86.Build.0 = Release|x86 + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection +EndGlobal diff --git a/src/CollectdWinService/Aggregator.cs b/src/CollectdWinService/Aggregator.cs new file mode 100644 index 0000000..6cc5c76 --- /dev/null +++ b/src/CollectdWinService/Aggregator.cs @@ -0,0 +1,181 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class CacheEntry + { + public MetricValue MetricRate; + public double[] RawValues; + + public CacheEntry(MetricValue metricValue) + { + MetricRate = metricValue; + RawValues = (double[]) metricValue.Values.Clone(); + } + } + + internal class Aggregator + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private readonly Dictionary _cache; + private readonly Object _cacheLock; + private readonly bool _storeRates; + private readonly int _timeoutSeconds; + + public Aggregator(int timeoutSeconds, bool storeRates) + { + _cache = new Dictionary(); + _cacheLock = new Object(); + _timeoutSeconds = timeoutSeconds; + _storeRates = storeRates; + } + + public void Aggregate(ref MetricValue metricValue) + { + // If rates are not stored then there is nothing to aggregate + if (!_storeRates) + { + return; + } + IList dsl = DataSetCollection.Instance.GetDataSource(metricValue.TypeName); + if (dsl == null || metricValue.Values.Length != dsl.Count) + { + return; + } + + double now = Util.GetNow(); + + lock (_cacheLock) + { + CacheEntry cEntry; + string key = metricValue.Key(); + + if (!(_cache.TryGetValue(key, out cEntry))) + { + cEntry = new CacheEntry(metricValue.DeepCopy()); + for (int i = 0; i < metricValue.Values.Length; i++) + { + if (dsl[i].Type == DsType.Derive || + dsl[i].Type == DsType.Absolute || + dsl[i].Type == DsType.Counter) + { + metricValue.Values[i] = double.NaN; + cEntry.MetricRate.Values[i] = double.NaN; + } + } + cEntry.MetricRate.Epoch = now; + _cache[key] = cEntry; + return; + } + for (int i = 0; i < metricValue.Values.Length; i++) + { + double rawValNew = metricValue.Values[i]; + double rawValOld = cEntry.RawValues[i]; + double rawValDiff = rawValNew - rawValOld; + double timeDiff = cEntry.MetricRate.Epoch - now; + + double rateNew = rawValDiff/timeDiff; + + switch (dsl[i].Type) + { + case DsType.Gauge: + // no rates calculations are done, values are stored as-is for gauge + cEntry.RawValues[i] = rawValNew; + cEntry.MetricRate.Values[i] = rawValNew; + break; + + case DsType.Absolute: + // similar to gauge, except value will be divided by time diff + cEntry.MetricRate.Values[i] = metricValue.Values[i]/timeDiff; + cEntry.RawValues[i] = rawValNew; + metricValue.Values[i] = cEntry.MetricRate.Values[i]; + break; + + case DsType.Derive: + cEntry.RawValues[i] = rawValNew; + cEntry.MetricRate.Values[i] = rateNew; + metricValue.Values[i] = rateNew; + + break; + + case DsType.Counter: + // Counters are very simlar to derive except when counter wraps around + if (rawValNew < rawValOld) + { + // counter has wrapped around + cEntry.MetricRate.Values[i] = metricValue.Values[i]/timeDiff; + cEntry.RawValues[i] = rawValNew; + metricValue.Values[i] = cEntry.MetricRate.Values[i]; + } + else + { + cEntry.MetricRate.Values[i] = rateNew; + cEntry.RawValues[i] = rawValNew; + metricValue.Values[i] = rateNew; + } + break; + } + + // range checks + if (metricValue.Values[i] < dsl[i].Min) + { + metricValue.Values[i] = dsl[i].Min; + cEntry.RawValues[i] = metricValue.Values[i]; + } + if (metricValue.Values[i] > dsl[i].Max) + { + metricValue.Values[i] = dsl[i].Max; + cEntry.RawValues[i] = metricValue.Values[i]; + } + + cEntry.MetricRate.Epoch = now; + _cache[key] = cEntry; + } + } + } + + public void RemoveExpiredEntries() + { + // If rates are not stored then there is nothing to remove + if (!_storeRates) + { + return; + } + double now = Util.GetNow(); + double expirationTime = now - _timeoutSeconds; + var removeList = new List(); + + lock (_cacheLock) + { + removeList.AddRange(from pair in _cache + let cEntry = pair.Value + where cEntry.MetricRate.Epoch < expirationTime + select pair.Key); + if (removeList.Count > 0) + Logger.Debug("Removing expired entries: {0}", removeList.Count); + foreach (string key in removeList) + { + _cache.Remove(key); + } + } + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/AmqpPlugin.cs b/src/CollectdWinService/AmqpPlugin.cs new file mode 100644 index 0000000..7fe8229 --- /dev/null +++ b/src/CollectdWinService/AmqpPlugin.cs @@ -0,0 +1,154 @@ +using System; +using System.Configuration; +using System.Text; +using NLog; +using RabbitMQ.Client; + +namespace BloombergFLP.CollectdWin +{ + internal class AmqpPlugin : IMetricsWritePlugin + { + private const int ConnectionRetryDelay = 60; // 1 minute + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private readonly Object _connectionLock; + private IModel _channel; + private bool _connected; + private IConnection _connection; + private string _exchange; + private double _lastConnectTime; + private string _routingKeyPrefix; + private string _url; + + public AmqpPlugin() + { + _connected = false; + _lastConnectTime = 0; + _connectionLock = new object(); + } + + public void Configure() + { + var config = ConfigurationManager.GetSection("CollectdWinConfig") as CollectdWinConfig; + if (config == null) + { + throw new Exception("Cannot get configuration section : CollectdWinConfig"); + } + + string user = config.Amqp.Publish.User; + string password = config.Amqp.Publish.Password; + string host = config.Amqp.Publish.Host; + int port = config.Amqp.Publish.Port; + string vhost = config.Amqp.Publish.VirtualHost; + + _url = "amqp://" + user + ":" + password + "@" + host + ":" + port + "/" + vhost; + _exchange = config.Amqp.Publish.Exchange; + _routingKeyPrefix = config.Amqp.Publish.RoutingKeyPrefix; + Logger.Info("Amqp plugin configured"); + } + + public void Start() + { + Logger.Trace("Start() begin."); + StartConnection(); + Logger.Info("Amqp plugin started"); + } + + public void Stop() + { + Logger.Trace("CloseConnection() begin"); + CloseConnection(); + Logger.Info("Amqp plugin stopped"); + } + + public void Write(MetricValue metric) + { + if (metric == null) + { + Logger.Debug("write() - Invalid null metric"); + return; + } + if (!_connected) + StartConnection(); + if (_connected && _channel != null) + { + string routingKey = GetAmqpRoutingKey(metric); + string message = metric.GetMetricJsonStr(); + try + { + _channel.BasicPublish(_exchange, routingKey, null, Encoding.UTF8.GetBytes(message)); + } + catch + { + CloseConnection(); + } + } + } + + public void StartConnection() + { + double now = Util.GetNow(); + if (now < (_lastConnectTime + ConnectionRetryDelay)) + { + return; + } + lock (_connectionLock) + { + try + { + var cf = new ConnectionFactory {Uri = _url}; + _connection = cf.CreateConnection(); + _channel = _connection.CreateModel(); + + _connected = true; + _lastConnectTime = Util.GetNow(); + Logger.Debug("Connection started."); + } + catch (Exception exp) + { + Logger.Error("Got exception when connecting to AMQP broker : ", exp); + } + } + } + + public void CloseConnection() + { + lock (_connectionLock) + { + try + { + _channel.Close(); + _connection.Close(); + Logger.Debug("Connection closed."); + } + catch (Exception exp) + { + Logger.Error("Got exception when closing AMQP connection : ", exp); + } + _connected = false; + } + } + + private string GetAmqpRoutingKey(MetricValue metric) + { + string routingKey = _routingKeyPrefix + "." + metric.HostName + "." + metric.PluginName + "." + + metric.PluginInstanceName + "." + metric.TypeName + "." + metric.TypeInstanceName; + + return (routingKey); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/CollectdWinService.cs b/src/CollectdWinService/CollectdWinService.cs new file mode 100644 index 0000000..2b5ded3 --- /dev/null +++ b/src/CollectdWinService/CollectdWinService.cs @@ -0,0 +1,64 @@ +using System.ServiceProcess; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + public class CollectdWinService : ServiceBase + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private MetricsCollector _metricsCollector; + + public CollectdWinService() + { + InitializeComponent(); + } + + protected override void OnStart(string[] args) + { + StartService(args); + } + + protected override void OnStop() + { + StopService(); + } + + private void InitializeComponent() + { + ServiceName = "Bloomberg Metrics Collector Service"; + } + + // public accessibility for running as a console application + public virtual void StartService(params string[] args) + { + Logger.Trace("StartService() begin"); + _metricsCollector = new MetricsCollector(); + _metricsCollector.ConfigureAll(); + _metricsCollector.StartAll(); + Logger.Trace("StartService() return"); + } + + // public accessibility for running as a console application + public virtual void StopService() + { + Logger.Trace("StopService() begin"); + _metricsCollector.StopAll(); + Logger.Trace("StopService() return"); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/CollectdWinService.csproj b/src/CollectdWinService/CollectdWinService.csproj new file mode 100644 index 0000000..83b3f17 --- /dev/null +++ b/src/CollectdWinService/CollectdWinService.csproj @@ -0,0 +1,173 @@ + + + + + Debug + AnyCPU + {D4244E6B-84EC-41A0-96A7-4F489F6098BC} + WinExe + Properties + BloombergFLP.CollectdWin + CollectdWinService + v3.5 + 512 + + false + publish\ + true + Disk + false + Foreground + 7 + Days + false + false + true + 2 + 1.0.0.%2a + false + true + true + + + AnyCPU + true + full + false + bin\Debug\ + DEBUG;TRACE + prompt + 4 + + + AnyCPU + pdbonly + true + bin\Release\ + TRACE + prompt + 4 + + + BF90C155A87E1D740767C321E58F1CEC4DC9C8C9 + + + MetricsCollectorService_TemporaryKey.pfx + + + false + + + true + + + true + bin\x64\Debug\ + DEBUG;TRACE + full + x64 + prompt + MinimumRecommendedRules.ruleset + + + bin\x64\Release\ + TRACE + true + pdbonly + x64 + prompt + MinimumRecommendedRules.ruleset + + + true + bin\x86\Debug\ + DEBUG;TRACE + full + x86 + prompt + MinimumRecommendedRules.ruleset + + + bin\x86\Release\ + TRACE + true + pdbonly + x86 + prompt + MinimumRecommendedRules.ruleset + + + LocalIntranet + + + + + ..\..\third-party\NLog.dll + + + ..\..\third-party\RabbitMQ.Client.dll + + + + + + + + + + + + + + + + + + Component + + + + + + + + + + + + + + + + CollectdWinService.cs + + + + + Designer + + + Always + + + + + False + .NET Framework 3.5 SP1 Client Profile + false + + + False + .NET Framework 3.5 SP1 + true + + + + + \ No newline at end of file diff --git a/src/CollectdWinService/CollectdWinService.resx b/src/CollectdWinService/CollectdWinService.resx new file mode 100644 index 0000000..73afb87 --- /dev/null +++ b/src/CollectdWinService/CollectdWinService.resx @@ -0,0 +1,123 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + text/microsoft-resx + + + 2.0 + + + System.Resources.ResXResourceReader, System.Windows.Forms, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + System.Resources.ResXResourceWriter, System.Windows.Forms, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + False + + \ No newline at end of file diff --git a/src/CollectdWinService/ConsolePlugin.cs b/src/CollectdWinService/ConsolePlugin.cs new file mode 100644 index 0000000..2f2e16e --- /dev/null +++ b/src/CollectdWinService/ConsolePlugin.cs @@ -0,0 +1,45 @@ +using System; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class ConsolePlugin : IMetricsWritePlugin + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + + public void Configure() + { + Logger.Info("console plugin configured"); + } + + public void Start() + { + Logger.Info("console plugin started"); + } + + public void Stop() + { + Logger.Info("console plugin stopped"); + } + + public void Write(MetricValue metric) + { + Console.WriteLine("ConsolePlugin: {0}", metric.GetMetricJsonStr()); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/Histogram.cs b/src/CollectdWinService/Histogram.cs new file mode 100644 index 0000000..941dda1 --- /dev/null +++ b/src/CollectdWinService/Histogram.cs @@ -0,0 +1,127 @@ +using System; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class Histogram + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private readonly int[] _bins; + private readonly int _numBins; + private int _binSize; + private double _max; + private double _min; + private int _num; + private double _sum; + + public Histogram() + { + _min = _max = _sum = _num = 0; + _numBins = 100; + _bins = new int[100]; + _binSize = 8; + } + + private int GetBin(double val) + { + var bin = (int) (val/_binSize); + return (bin); + } + + public void Resize(double val) + { + double requiredBinSize = val/_numBins; + // to reduce frequent resizing, new bin size will be the the next nearest power of 2 + // eg: 16, 32, 64, 128, 256, 512, 1024, 2048, 5086 + var newBinSize = (int) Math.Pow(2, Math.Ceiling(Math.Log(requiredBinSize, 2))); + int oldBinSize = _binSize; + for (int i = 1; i < _numBins; i++) + { + val = i*oldBinSize; + int newBin = (int) val/newBinSize; + if (i == newBin) + continue; + _bins[newBin] += _bins[i]; + _bins[i] = 0; + } + _binSize = newBinSize; + Logger.Debug("Resize() - OldBinSize:{0} has been replaced with the NewBinSize:{1}", oldBinSize, newBinSize); + } + + public void AddValue(double val) + { + int bin = GetBin(val); + if (bin >= _numBins) + { + Resize(val); + bin = GetBin(val); + Logger.Debug("Got new bin:{0}", bin); + } + _bins[bin]++; + + _min = (_min > val) ? val : _min; + _max = (_max < val) ? val : _max; + _sum += val; + _num++; + } + + public double GetPercentile(float percent) + { + double percentUpper = 0; + double percentLower = 0; + double sum = 0; + + if (percent < 0 || percent > 100 || _num <= 0) + return (0); + + int i; + for (i = 0; i < _numBins; i++) + { + percentLower = percentUpper; + sum += _bins[i]; + percentUpper = 100*(sum/_num); + if (percentUpper >= percent) + break; + } + if (Math.Abs(percentUpper) < 0.01 || i >= _numBins) + return (0); + double valLower = i*_binSize; + double valUpper = (i + 1)*_binSize; + + double val = (((percentUpper - percent)*valLower) + ((percent - percentLower)*valUpper))/ + (percentUpper - percentLower); + + return (val); + } + + public void Reset() + { + _min = _max = _sum = _num = 0; + for (int i = 0; i < _numBins; i++) + _bins[i] = 0; + } + + public override string ToString() + { + String logstr = String.Format("Min:{0} Max:{1} Sum{2} Num{3}", _min, _max, _sum, _num); + for (int i = 0; i < _numBins; i++) + logstr += ", " + _bins[i]; + return (logstr); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/MetricsCollector.cs b/src/CollectdWinService/MetricsCollector.cs new file mode 100644 index 0000000..66f4a78 --- /dev/null +++ b/src/CollectdWinService/MetricsCollector.cs @@ -0,0 +1,226 @@ +using System; +using System.Collections.Generic; +using System.Configuration; +using System.Linq; +using System.Threading; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class MetricsCollector + { + private const int MaxQueueSize = 30000; + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private readonly Aggregator _aggregator; + private readonly int _interval; + private readonly Queue _metricValueQueue; + private readonly IList _plugins; + private readonly Object _queueLock; + private readonly int _timeout; + private Thread _aggregatorThread; + private Thread _readThread; + private bool _runAggregatorThread; + private bool _runReadThread, _runWriteThread; + private Thread _writeThread; + + public MetricsCollector() + { + var config = ConfigurationManager.GetSection("CollectdWinConfig") as CollectdWinConfig; + if (config == null) + { + Logger.Error("Cannot get configuration section"); + return; + } + + _runReadThread = false; + _runWriteThread = false; + + var registry = new PluginRegistry(); + _plugins = registry.CreatePlugins(); + + _interval = config.GeneralSettings.Interval; + if (_interval <= 10) + _interval = 10; + + _timeout = config.GeneralSettings.Timeout; + if (_timeout <= _interval) + _timeout = _interval*3; + + bool storeRates = config.GeneralSettings.StoreRates; + + _aggregator = new Aggregator(_timeout, storeRates); + + _metricValueQueue = new Queue(); + _queueLock = new Object(); + } + + public void ConfigureAll() + { + Logger.Trace("ConfigureAll() begin"); + foreach (IMetricsPlugin plugin in _plugins) + plugin.Configure(); + Logger.Trace("ConfigureAll() return"); + } + + public void StartAll() + { + Logger.Trace("StartAll() begin"); + foreach (IMetricsPlugin plugin in _plugins) + plugin.Start(); + + _runWriteThread = true; + _writeThread = new Thread(WriteThreadProc); + _writeThread.Start(); + + _runReadThread = true; + _readThread = new Thread(ReadThreadProc); + _readThread.Start(); + + _runAggregatorThread = true; + _aggregatorThread = new Thread(AggregatorThreadProc); + _aggregatorThread.Start(); + Logger.Trace("StartAll() return"); + } + + public void StopAll() + { + Logger.Trace("StopAll() begin"); + _runReadThread = false; + _runWriteThread = false; + _runAggregatorThread = false; + + _readThread.Interrupt(); + _writeThread.Interrupt(); + _aggregatorThread.Interrupt(); + + foreach (IMetricsPlugin plugin in _plugins) + plugin.Stop(); + Logger.Trace("StopAll() end"); + } + + private void ReadThreadProc() + { + Logger.Trace("ReadThreadProc() begin"); + int numMetricsDropped = 0; + while (_runReadThread) + { + try + { + foreach (IMetricsPlugin plugin in _plugins) + { + var readPlugin = plugin as IMetricsReadPlugin; + if (readPlugin == null) + { + // skip if plugin is not a readplugin, it might be a writeplugin + continue; + } + IList metricValues = readPlugin.Read(); + if (metricValues == null || !metricValues.Any()) + continue; + lock (_queueLock) + { + foreach (MetricValue metric in metricValues) + { + _metricValueQueue.Enqueue(metric); + while (_metricValueQueue.Count >= MaxQueueSize) + { + // When queue size grows above the Max limit, + // old entries are removed + _metricValueQueue.Dequeue(); + if ((++numMetricsDropped%1000) == 0) + { + Logger.Error( + "Number of metrics dropped : {0}", + numMetricsDropped); + } + } + } + } + } + Thread.Sleep(_interval*1000); + } + catch (Exception exp) + { + Logger.Error("ReadThreadProc() got exception : ", exp); + } + } + Logger.Trace("ReadThreadProc() return"); + } + + private void WriteThreadProc() + { + Logger.Trace("WriteThreadProc() begin"); + while (_runWriteThread) + { + try + { + while (_metricValueQueue.Count > 0) + { + MetricValue metricValue = null; + lock (_queueLock) + { + if (_metricValueQueue.Count > 0) + metricValue = _metricValueQueue.Dequeue(); + } + if (metricValue != null) + { + metricValue.Interval = _interval; + + _aggregator.Aggregate(ref metricValue); + + foreach (IMetricsPlugin plugin in _plugins) + { + var writePlugin = plugin as IMetricsWritePlugin; + if (writePlugin == null) + { + // skip if plugin is not a writeplugin, it might be a readplugin + continue; + } + writePlugin.Write(metricValue); + } + } + } + Thread.Sleep(_interval*1000); + } + catch (Exception exp) + { + Logger.Error("WriteThreadProc() got exception : ", exp); + } + } + Logger.Trace("WriteThreadProc() return"); + } + + private void AggregatorThreadProc() + { + Logger.Trace("AggregatorThreadProc() begin"); + while (_runAggregatorThread) + { + try + { + _aggregator.RemoveExpiredEntries(); + Thread.Sleep(_timeout*1000); + } + catch (Exception exp) + { + Logger.Error("AggregatorThreadProc() got exception : ", exp); + } + } + Logger.Trace("AggregatorThreadProc() return"); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/MetricsConfiguration.cs b/src/CollectdWinService/MetricsConfiguration.cs new file mode 100644 index 0000000..9b88a0b --- /dev/null +++ b/src/CollectdWinService/MetricsConfiguration.cs @@ -0,0 +1,424 @@ +using System; +using System.Configuration; + +namespace BloombergFLP.CollectdWin +{ + internal class CollectdWinConfig : ConfigurationSection + { + [ConfigurationProperty("GeneralSettings", IsRequired = false)] + public GeneralSettingsConfig GeneralSettings + { + get { return (GeneralSettingsConfig) base["GeneralSettings"]; } + set { base["GeneralSettings"] = value; } + } + + [ConfigurationProperty("PluginRegistry", IsRequired = true)] + [ConfigurationCollection(typeof (PluginCollection), AddItemName = "Plugin")] + public PluginCollection PluginRegistry + { + get { return (PluginCollection) base["PluginRegistry"]; } + set { base["PluginRegistry"] = value; } + } + + [ConfigurationProperty("Statsd", IsRequired = false)] + public StatsdConfig Statsd + { + get { return (StatsdConfig) base["Statsd"]; } + set { base["Statsd"] = value; } + } + + [ConfigurationProperty("Amqp", IsRequired = false)] + public AmqpConfig Amqp + { + get { return (AmqpConfig) base["Amqp"]; } + set { base["Amqp"] = value; } + } + + [ConfigurationProperty("WindowsPerformanceCounters", IsRequired = false)] + public WindowsPerformanceCountersConfig WindowsPerformanceCounters + { + get { return (WindowsPerformanceCountersConfig) base["WindowsPerformanceCounters"]; } + set { base["WindowsPerformanceCounters"] = value; } + } + + public static CollectdWinConfig GetConfig() + { + return (CollectdWinConfig) ConfigurationManager.GetSection("CollectdWinConfig") ?? new CollectdWinConfig(); + } + + public sealed class AmqpConfig : ConfigurationElement + { + [ConfigurationProperty("Publish", IsRequired = false)] + public PublishConfig Publish + { + get { return (PublishConfig) base["Publish"]; } + set { base["Publish"] = value; } + } + + public sealed class PublishConfig : ConfigurationElement + { + [ConfigurationProperty("Name", IsRequired = true)] + public string Name + { + get { return (string) base["Name"]; } + set { base["Name"] = value; } + } + + [ConfigurationProperty("Host", IsRequired = false)] + public string Host + { + get { return (string) base["Host"]; } + set { base["Host"] = value; } + } + + [ConfigurationProperty("Port", IsRequired = true)] + public int Port + { + get { return (int) base["Port"]; } + set { base["Port"] = value; } + } + + [ConfigurationProperty("VirtualHost", IsRequired = false)] + public string VirtualHost + { + get { return (string) base["VirtualHost"]; } + set { base["VirtualHost"] = value; } + } + + [ConfigurationProperty("User", IsRequired = false)] + public string User + { + get { return (string) base["User"]; } + set { base["User"] = value; } + } + + [ConfigurationProperty("Password", IsRequired = false)] + public string Password + { + get { return (string) base["Password"]; } + set { base["Password"] = value; } + } + + [ConfigurationProperty("Exchange", IsRequired = false)] + public string Exchange + { + get { return (string) base["Exchange"]; } + set { base["Exchange"] = value; } + } + + [ConfigurationProperty("RoutingKeyPrefix", IsRequired = false)] + public string RoutingKeyPrefix + { + get { return (string) base["RoutingKeyPrefix"]; } + set { base["RoutingKeyPrefix"] = value; } + } + } + } + + public sealed class CounterConfig : ConfigurationElement + { + [ConfigurationProperty("Category", IsRequired = true)] + public string Category + { + get { return (string) base["Category"]; } + set { base["Category"] = value; } + } + + [ConfigurationProperty("Name", IsRequired = true)] + public string Name + { + get { return (string) base["Name"]; } + set { base["Name"] = value; } + } + + + [ConfigurationProperty("Instance", IsRequired = false)] + public string Instance + { + get { return (string) base["Instance"]; } + set { base["Instance"] = value; } + } + + [ConfigurationProperty("CollectdPlugin", IsRequired = true)] + public string CollectdPlugin + { + get { return (string) base["CollectdPlugin"]; } + set { base["CollectdPlugin"] = value; } + } + + [ConfigurationProperty("CollectdPluginInstance", IsRequired = false)] + public string CollectdPluginInstance + { + get { return (string) base["CollectdPluginInstance"]; } + set { base["CollectdPluginInstance"] = value; } + } + + [ConfigurationProperty("CollectdType", IsRequired = true)] + public string CollectdType + { + get { return (string) base["CollectdType"]; } + set { base["CollectdType"] = value; } + } + + [ConfigurationProperty("CollectdTypeInstance", IsRequired = true)] + public string CollectdTypeInstance + { + get { return (string) base["CollectdTypeInstance"]; } + set { base["CollectdTypeInstance"] = value; } + } + + [ConfigurationProperty("ScaleUpFactor", IsRequired = false)] + public uint ScaleUpFactor + { + get { return (uint) base["ScaleUpFactor"]; } + set { base["ScaleUpFactor"] = value; } + } + + [ConfigurationProperty("ScaleDownFactor", IsRequired = false)] + public uint ScaleDownFactor + { + get { return (uint) base["ScaleDownFactor"]; } + set { base["ScaleDownFactor"] = value; } + } + } + + public class CounterConfigCollection : ConfigurationElementCollection + { + protected override ConfigurationElement CreateNewElement() + { + return new CounterConfig(); + } + + protected override object GetElementKey(ConfigurationElement element) + { + var counterConfig = (CounterConfig) element; + return (counterConfig.Category + "_" + counterConfig.Name + "_" + counterConfig.Instance); + } + } + + public sealed class GeneralSettingsConfig : ConfigurationElement + { + [ConfigurationProperty("Interval", IsRequired = true)] + public int Interval + { + get { return (int) base["Interval"]; } + set { base["Interval"] = value; } + } + + [ConfigurationProperty("Timeout", IsRequired = true)] + public int Timeout + { + get { return (int) base["Timeout"]; } + set { base["Timeout"] = value; } + } + + [ConfigurationProperty("StoreRates", IsRequired = true)] + public bool StoreRates + { + get { return (bool) base["StoreRates"]; } + set { base["StoreRates"] = value; } + } + } + + public class PluginCollection : ConfigurationElementCollection + { + protected override ConfigurationElement CreateNewElement() + { + return new PluginConfig(); + } + + protected override object GetElementKey(ConfigurationElement element) + { + return (((PluginConfig) element).UniqueId); + } + } + + public sealed class PluginConfig : ConfigurationElement + { + public PluginConfig() + { + UniqueId = Guid.NewGuid(); + } + + internal Guid UniqueId { get; set; } + + [ConfigurationProperty("Name", IsRequired = true)] + public string Name + { + get { return (string) base["Name"]; } + set { base["Name"] = value; } + } + + [ConfigurationProperty("Class", IsRequired = true)] + public string Class + { + get { return (string) base["Class"]; } + set { base["Class"] = value; } + } + + [ConfigurationProperty("Enable", IsRequired = true)] + public bool Enable + { + get { return (bool) base["Enable"]; } + set { base["Enable"] = value; } + } + } + + public sealed class StatsdConfig : ConfigurationElement + { + [ConfigurationProperty("Host", IsRequired = false)] + public string Host + { + get { return (string) base["Host"]; } + set { base["Host"] = value; } + } + + [ConfigurationProperty("Port", IsRequired = true)] + public int Port + { + get { return (int) base["Port"]; } + set { base["Port"] = value; } + } + + [ConfigurationProperty("DeleteCache", IsRequired = true)] + public DeleteCacheConfig DeleteCache + { + get { return (DeleteCacheConfig) base["DeleteCache"]; } + set { base["DeleteCache"] = value; } + } + + [ConfigurationProperty("Timer", IsRequired = true)] + public TimerConfig Timer + { + get { return (TimerConfig) base["Timer"]; } + set { base["Timer"] = value; } + } + + public sealed class DeleteCacheConfig : ConfigurationElement + { + [ConfigurationProperty("Counters", IsRequired = true)] + public bool Counters + { + get { return (bool) base["Counters"]; } + set { base["Counters"] = value; } + } + + [ConfigurationProperty("Timers", IsRequired = true)] + public bool Timers + { + get { return (bool) base["Timers"]; } + set { base["Timers"] = value; } + } + + [ConfigurationProperty("Gauges", IsRequired = true)] + public bool Gauges + { + get { return (bool) base["Gauges"]; } + set { base["Gauges"] = value; } + } + + [ConfigurationProperty("Sets", IsRequired = true)] + public bool Sets + { + get { return (bool) base["Sets"]; } + set { base["Sets"] = value; } + } + } + + public class PercentileCollection : ConfigurationElementCollection + { + protected override ConfigurationElement CreateNewElement() + { + return new PercentileConfig(); + } + + protected override object GetElementKey(ConfigurationElement element) + { + return (((PercentileConfig) element).UniqueId); + } + } + + public sealed class PercentileConfig : ConfigurationElement + { + public PercentileConfig() + { + UniqueId = Guid.NewGuid(); + } + + internal Guid UniqueId { get; set; } + + [ConfigurationProperty("Value", IsRequired = true)] + public float Value + { + get { return (float) base["Value"]; } + set { base["Value"] = value; } + } + } + + public sealed class TimerConfig : ConfigurationElement + { + [ConfigurationProperty("Lower", IsRequired = true)] + public bool Lower + { + get { return (bool) base["Lower"]; } + set { base["Lower"] = value; } + } + + [ConfigurationProperty("Upper", IsRequired = true)] + public bool Upper + { + get { return (bool) base["Upper"]; } + set { base["Upper"] = value; } + } + + [ConfigurationProperty("Sum", IsRequired = true)] + public bool Sum + { + get { return (bool) base["Sum"]; } + set { base["Sum"] = value; } + } + + [ConfigurationProperty("Count", IsRequired = true)] + public bool Count + { + get { return (bool) base["Count"]; } + set { base["Count"] = value; } + } + + [ConfigurationProperty("Percentiles", IsRequired = false)] + [ConfigurationCollection(typeof (PercentileCollection), AddItemName = "Percentile")] + public PercentileCollection Percentiles + { + get { return (PercentileCollection) base["Percentiles"]; } + set { base["Percentiles"] = value; } + } + } + } + + public sealed class WindowsPerformanceCountersConfig : ConfigurationElement + { + [ConfigurationProperty("Counters", IsRequired = false)] + [ConfigurationCollection(typeof (CounterConfigCollection), AddItemName = "Counter")] + public CounterConfigCollection Counters + { + get { return (CounterConfigCollection) base["Counters"]; } + set { base["Counters"] = value; } + } + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/MetricsPlugin.cs b/src/CollectdWinService/MetricsPlugin.cs new file mode 100644 index 0000000..9d9aae6 --- /dev/null +++ b/src/CollectdWinService/MetricsPlugin.cs @@ -0,0 +1,103 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal interface IMetricsPlugin + { + void Configure(); + void Start(); + void Stop(); + } + + internal interface IMetricsReadPlugin : IMetricsPlugin + { + IList Read(); + } + + internal interface IMetricsWritePlugin : IMetricsPlugin + { + void Write(MetricValue metric); + } + + internal class MetricValue + { + private const string MetricJsonFormat = + @"{{""host"":""{0}"", ""plugin"":""{1}"", ""plugin_instance"":""{2}""," + + @" ""type"":""{3}"", ""type_instance"":""{4}"", ""time"":{5}, ""interval"":{6}," + + @" ""dstypes"":[{7}], ""dsnames"":[{8}], ""values"":[{9}]}}"; + + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + + public string HostName { get; set; } + public string PluginName { get; set; } + public string PluginInstanceName { get; set; } + public string TypeName { get; set; } + public string TypeInstanceName { get; set; } + + public int Interval { get; set; } + public double Epoch { get; set; } + public double[] Values { get; set; } + + public string Key() + { + return (HostName + "." + PluginName + "." + PluginInstanceName + "." + TypeName + "." + TypeInstanceName); + } + + public MetricValue DeepCopy() + { + var other = (MetricValue) MemberwiseClone(); + other.HostName = String.Copy(HostName); + other.PluginName = String.Copy(PluginName); + other.PluginInstanceName = String.Copy(PluginInstanceName); + other.TypeName = String.Copy(TypeName); + other.TypeInstanceName = String.Copy(TypeInstanceName); + other.Values = (double[]) Values.Clone(); + return (other); + } + + public string GetMetricJsonStr() + { + IList dsList = DataSetCollection.Instance.GetDataSource(TypeName); + var dsNames = new List(); + var dsTypes = new List(); + if (dsList == null) + { + Logger.Debug("Invalid type : {0}, not found in types.db", TypeName); + } + else + { + foreach (DataSource ds in dsList) + { + dsNames.Add(ds.Name); + dsTypes.Add(ds.Type.ToString().ToLower()); + } + } + string dsTypesStr = string.Join(",", dsTypes.ConvertAll(m => string.Format("\"{0}\"", m)).ToArray()); + string dsNamesStr = string.Join(",", dsNames.ConvertAll(m => string.Format("\"{0}\"", m)).ToArray()); + string valStr = string.Join(",", Array.ConvertAll(Values, val => val.ToString(CultureInfo.InvariantCulture))); + + string res = string.Format(MetricJsonFormat, HostName, PluginName, + PluginInstanceName, TypeName, TypeInstanceName, Epoch, + Interval, dsTypesStr, dsNamesStr, valStr); + return (res); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/PluginRegistry.cs b/src/CollectdWinService/PluginRegistry.cs new file mode 100644 index 0000000..bd78692 --- /dev/null +++ b/src/CollectdWinService/PluginRegistry.cs @@ -0,0 +1,64 @@ +using System; +using System.Collections.Generic; +using System.Configuration; +using System.Linq; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class PluginRegistry + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + + private readonly Dictionary _registry = new Dictionary(); + + public PluginRegistry() + { + var config = ConfigurationManager.GetSection("CollectdWinConfig") as CollectdWinConfig; + if (config == null) + { + Logger.Error("Cannot get configuration section"); + return; + } + foreach ( + CollectdWinConfig.PluginConfig pluginConfig in + config.PluginRegistry.Cast() + .Where(pluginConfig => pluginConfig.Enable)) + { + _registry[pluginConfig.Name] = pluginConfig.Class; + } + } + + public IList CreatePlugins() + { + IList plugins = new List(); + foreach (var entry in _registry) + { + Type classType = Type.GetType(entry.Value); + if (classType == null) + { + Logger.Error("Cannot create plugin:{0}, class:{1}", entry.Key, entry.Value); + continue; + } + var plugin = (IMetricsPlugin) Activator.CreateInstance(classType); + plugins.Add(plugin); + } + return (plugins); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/Program.cs b/src/CollectdWinService/Program.cs new file mode 100644 index 0000000..68911d4 --- /dev/null +++ b/src/CollectdWinService/Program.cs @@ -0,0 +1,54 @@ +using System; +using System.Configuration; +using System.ServiceProcess; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + public static class Program + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + + /// + /// The main entry point for the application. + /// + public static void Main(string[] args) + { + var config = ConfigurationManager.GetSection("CollectdWinConfig") as CollectdWinConfig; + if (config == null) + { + Logger.Error("Main(): cannot get configuration section"); + return; + } + + var collectdWinService = new CollectdWinService(); + + if (Array.Find(args, s => s.Equals(@"/console")) != null) + { + // run as a console application for testing and debugging purpose + collectdWinService.StartService(); + } + else + { + // run as a windows service + ServiceBase[] servicesToRun = {collectdWinService}; + ServiceBase.Run(servicesToRun); + } + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/Properties/AssemblyInfo.cs b/src/CollectdWinService/Properties/AssemblyInfo.cs new file mode 100644 index 0000000..94e3c02 --- /dev/null +++ b/src/CollectdWinService/Properties/AssemblyInfo.cs @@ -0,0 +1,39 @@ +using System.Reflection; +using System.Runtime.InteropServices; + +// General Information about an assembly is controlled through the following +// set of attributes. Change these attribute values to modify the information +// associated with an assembly. + +[assembly: AssemblyTitle("CollectdWinService")] +[assembly: AssemblyDescription("")] +[assembly: AssemblyConfiguration("")] +[assembly: AssemblyCompany("Bloomberg LP")] +[assembly: AssemblyProduct("CollectdWinService")] +[assembly: AssemblyCopyright("Copyright © Bloomberg LP 2015")] +[assembly: AssemblyTrademark("")] +[assembly: AssemblyCulture("")] + +// Setting ComVisible to false makes the types in this assembly not visible +// to COM components. If you need to access a type in this assembly from +// COM, set the ComVisible attribute to true on that type. + +[assembly: ComVisible(false)] + +// The following GUID is for the ID of the typelib if this project is exposed to COM + +[assembly: Guid("dc0404f4-acd7-40b3-ab7a-6e63f023ac40")] + +// Version information for an assembly consists of the following four values: +// +// Major Version +// Minor Version +// Build Number +// Revision +// +// You can specify all the values or you can default the Build and Revision Numbers +// by using the '*' as shown below: +// [assembly: AssemblyVersion("1.0.*")] + +[assembly: AssemblyVersion("0.5.1.0")] +[assembly: AssemblyFileVersion("0.5.1.0")] \ No newline at end of file diff --git a/src/CollectdWinService/StatsdAggregator.cs b/src/CollectdWinService/StatsdAggregator.cs new file mode 100644 index 0000000..d344d76 --- /dev/null +++ b/src/CollectdWinService/StatsdAggregator.cs @@ -0,0 +1,191 @@ +using System; +using System.Collections.Generic; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class StatsdAggregator + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + + private readonly bool _delCounter; + private readonly bool _delGauge; + private readonly bool _delSet; + private readonly bool _delTimer; + private readonly string _hostName; + private readonly Object _lock; + private readonly Dictionary _metrics; + private readonly float[] _percentiles; + private readonly bool _timerCount; + private readonly bool _timerLower; + private readonly bool _timerSum; + private readonly bool _timerUpper; + + public StatsdAggregator(bool delCounter, bool delTimer, bool delGauge, bool delSet, bool timerLower, + bool timerUpper, bool timerSum, bool timerCount, float[] percentiles) + { + _lock = new Object(); + _metrics = new Dictionary(); + _hostName = Util.GetHostName(); + + _delCounter = delCounter; + _delTimer = delTimer; + _delGauge = delGauge; + _delSet = delSet; + + _timerLower = timerLower; + _timerUpper = timerUpper; + _timerSum = timerSum; + _timerCount = timerCount; + + _percentiles = percentiles; + + if (_percentiles != null && _percentiles.Length > 0) + StatsdMetric.Latency.HistogramEnabled = true; + else + StatsdMetric.Latency.HistogramEnabled = false; + + Logger.Info( + "Statsd config - delCounter:{0}, delTimer:{1}, delGauge:{2}, delSet:{3}, isHistogramEnabled:{4}", + _delCounter, _delTimer, _delGauge, _delSet, StatsdMetric.Latency.HistogramEnabled); + } + + public void AddMetric(StatsdMetric metric) + { + string key = metric.Type + "_" + metric.Name; + lock (_lock) + { + StatsdMetric oldMetric; + if (_metrics.TryGetValue(key, out oldMetric)) + { + //merge + double val = metric.Value; + oldMetric.AddValue(val); + } + else + { + _metrics[key] = metric; + } + } + } + + public IList Read() + { + lock (_lock) + { + var res = new List(); + var removeList = new List(); + foreach (var pair in _metrics) + { + StatsdMetric metric = pair.Value; + if (metric.NumUpdates <= 0 && + ((_delCounter && metric.Type == StatsdMetric.StatsdType.StatsdCounter) || + (_delTimer && metric.Type == StatsdMetric.StatsdType.StatsdTimer) || + (_delGauge && metric.Type == StatsdMetric.StatsdType.StatsdGauge) || + (_delSet && metric.Type == StatsdMetric.StatsdType.StatsdSet))) + { + removeList.Add(pair.Key); + continue; + } + var metricVal = new MetricValue + { + HostName = _hostName, + PluginName = "statsd", + PluginInstanceName = "", + TypeInstanceName = metric.Name, + Values = new[] {metric.GetMetric()} + }; + switch (metric.Type) + { + case StatsdMetric.StatsdType.StatsdGauge: + metricVal.TypeName = "gauge"; + break; + case StatsdMetric.StatsdType.StatsdTimer: + metricVal.TypeName = "latency"; + metricVal.TypeInstanceName += "-average"; + break; + case StatsdMetric.StatsdType.StatsdSet: + metricVal.TypeName = "objects"; + break; + default: + metricVal.TypeName = "derive"; + break; + } + TimeSpan t = DateTime.UtcNow - new DateTime(1970, 1, 1); + double epoch = t.TotalMilliseconds/1000; + metricVal.Epoch = Math.Round(epoch, 3); + + res.Add(metricVal); + + if (metric.Type == StatsdMetric.StatsdType.StatsdTimer) + { + if (_timerLower) + { + MetricValue lowerValue = metricVal.DeepCopy(); + lowerValue.TypeInstanceName = metric.Name + "-lower"; + lowerValue.Values[0] = metric.Lat.Min; + res.Add(lowerValue); + } + if (_timerUpper) + { + MetricValue upperValue = metricVal.DeepCopy(); + upperValue.TypeInstanceName = metric.Name + "-upper"; + upperValue.Values[0] = metric.Lat.Max; + res.Add(upperValue); + } + + if (_timerSum) + { + MetricValue upperSum = metricVal.DeepCopy(); + upperSum.TypeInstanceName = metric.Name + "-Sum"; + upperSum.Values[0] = metric.Lat.Sum; + res.Add(upperSum); + } + if (_timerCount) + { + MetricValue upperCount = metricVal.DeepCopy(); + upperCount.TypeInstanceName = metric.Name + "-count"; + upperCount.Values[0] = metric.Lat.Num; + res.Add(upperCount); + } + Histogram histogram = metric.Lat.Histogram; + if (_percentiles != null && _percentiles.Length > 0 && histogram != null) + { + foreach (float percentile in _percentiles) + { + double val = histogram.GetPercentile(percentile); + + MetricValue mv = metricVal.DeepCopy(); + mv.TypeInstanceName = metric.Name + "-percentile-" + percentile; + mv.Values[0] = val; + res.Add(mv); + } + } + } + metric.Reset(); + } + Logger.Debug("Removing entries that were not updated:{0}", removeList.Count); + foreach (string key in removeList) + { + _metrics.Remove(key); + } + return (res); + } + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/StatsdListener.cs b/src/CollectdWinService/StatsdListener.cs new file mode 100644 index 0000000..9057d90 --- /dev/null +++ b/src/CollectdWinService/StatsdListener.cs @@ -0,0 +1,117 @@ +using System; +using System.Net; +using System.Net.Sockets; +using System.Text; +using System.Threading; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class StatsdListener + { + public delegate void HandleMessage(string message); + + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + + private readonly IPEndPoint _endPoint; + + private readonly HandleMessage _messageHandler; + private readonly Socket _socket; + private bool _run; + + public StatsdListener(int port, HandleMessage handleMessage) + { + _messageHandler = handleMessage; + _endPoint = new IPEndPoint(IPAddress.Any, port); + _socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp); + } + + private void BindSocket() + { + while (!_socket.IsBound) + { + try + { + _socket.Bind(_endPoint); + } + catch (Exception exp) + { + Logger.Error("BindSocket failed: ", exp); + } + if (_socket.IsBound) + break; + Thread.Sleep(10*1000); + } + } + + private void CloseSocket() + { + try + { + _socket.Shutdown(SocketShutdown.Both); + _socket.Close(); + } + catch (Exception exp) + { + Logger.Error("CloseSocket failed: ", exp); + } + } + + public void Start() + { + Logger.Trace("Start() begin"); + var buffer = new byte[1024*4]; + + var sender = new IPEndPoint(IPAddress.Any, 0); + EndPoint remote = sender; + + _run = true; + while (_run) + { + if (!_socket.IsBound) + { + BindSocket(); + } + try + { + int recv = _socket.ReceiveFrom(buffer, ref remote); + string str = Encoding.ASCII.GetString(buffer, 0, recv); + str = str.TrimEnd('\r', '\n'); + + _messageHandler(str); + } + catch + { + CloseSocket(); + } + } + + Logger.Trace("Start() end"); + } + + public void Stop() + { + Logger.Trace("Stop() begin"); + _run = false; + // closing socket will cause Socket.ReceiveFrom() blocked call to + // throw SocketException, a work-around for shutting down a listener. + CloseSocket(); + Logger.Trace("Stop() end"); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/StatsdMetrics.cs b/src/CollectdWinService/StatsdMetrics.cs new file mode 100644 index 0000000..9cc0ee2 --- /dev/null +++ b/src/CollectdWinService/StatsdMetrics.cs @@ -0,0 +1,262 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class StatsdMetric + { + public enum StatsdType + { + StatsdCounter, + StatsdTimer, + StatsdGauge, + StatsdSet + }; + + private readonly HashSet _set; + + public StatsdMetric(string name, StatsdType type, double val) + { + Name = name; + Type = type; + _set = null; + Lat = null; + Value = val; + NumUpdates = 1; + + switch (Type) + { + case StatsdType.StatsdSet: + _set = new HashSet {val}; + break; + case StatsdType.StatsdTimer: + Lat = new Latency(); + Lat.Min = Lat.Max = Lat.Sum = val; + Lat.Num = 1; + break; + } + } + + public string Name { get; private set; } + public StatsdType Type { get; private set; } + public double Value { get; private set; } + public Latency Lat { get; private set; } + public int NumUpdates { get; private set; } + + public double GetMetric() + { + switch (Type) + { + case StatsdType.StatsdSet: + return (_set.Count); + + case StatsdType.StatsdTimer: + return (Lat.Sum/Lat.Num); + } + return (Value); + } + + public void Reset() + { + NumUpdates = 0; + + if (Type == StatsdType.StatsdSet && _set != null) + { + _set.Clear(); + } + if (Type == StatsdType.StatsdTimer && Lat != null) + { + Lat.Reset(); + } + } + + public void AddValue(double value) + { + switch (Type) + { + case StatsdType.StatsdCounter: + Value += value; + break; + case StatsdType.StatsdSet: + _set.Add(value); + break; + case StatsdType.StatsdGauge: + Value = value; + break; + case StatsdType.StatsdTimer: + Lat.AddValue(value); + break; + } + NumUpdates++; + } + + public class Latency + { + public double Max; + public double Min; + public int Num; + public double Sum; + + public Latency() + { + Min = Max = Sum = 0; + Num = 0; + Histogram = null; + if (HistogramEnabled) + { + Histogram = new Histogram(); + } + } + + public Histogram Histogram { get; private set; } + + public static bool HistogramEnabled { get; set; } + + public void AddValue(double val) + { + Min = (val < Min) ? val : Min; + Max = (val > Max) ? val : Max; + Sum += val; + Num++; + if (HistogramEnabled) + { + Histogram.AddValue(val); + } + } + + public void Reset() + { + Min = Max = Sum = 0; + Num = 0; + if (HistogramEnabled) + { + Histogram.Reset(); + } + } + }; + } + + internal class StatsdMetricParser + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + + public static IList Parse(string str) + { + var metrics = new List(); + string[] lines = str.Split('\n'); + metrics.AddRange(lines.Select(ParseLine).Where(metric => metric != null)); + return (metrics); + } + + /* + * StatsD metric collection protocol + * - metrics are separated by newlines + * - each line ar generally of the form: + * :| + * ** Gauges : :|g + * ** Counters : :|c[|@] + * ** Timers : :|ms + * ** Sets : :|s + */ + + public static StatsdMetric ParseLine(string line) + { + const string pattern = @"^(?.*):(?.*)\|(?.*)(\|\@(?.*))?$"; + var regex = new Regex(pattern, RegexOptions.IgnoreCase | RegexOptions.RightToLeft); + Match match = regex.Match(line); + + if (!match.Success) + { + Logger.Debug("Parser: Invalid statsd format [{0}]", line); + return (null); + } + GroupCollection groups = match.Groups; + + string name = groups["name"].Value; + string valstr = groups["value"].Value; + string typestr = groups["type"].Value; + string ratestr = groups["rate"].Value; + + if (String.IsNullOrEmpty(name) || String.IsNullOrEmpty(valstr) || String.IsNullOrEmpty(typestr)) + { + Logger.Debug("Parser: name/value/type are not optional [{0}]", line); + return (null); + } + StatsdMetric.StatsdType type; + + if (String.Compare(typestr, "g", StringComparison.OrdinalIgnoreCase) == 0) + type = StatsdMetric.StatsdType.StatsdGauge; + else if (String.Compare(typestr, "c", StringComparison.OrdinalIgnoreCase) == 0) + type = StatsdMetric.StatsdType.StatsdCounter; + else if (String.Compare(typestr, "s", StringComparison.OrdinalIgnoreCase) == 0) + type = StatsdMetric.StatsdType.StatsdSet; + else if (String.Compare(typestr, "ms", StringComparison.OrdinalIgnoreCase) == 0) + type = StatsdMetric.StatsdType.StatsdTimer; + else + { + Logger.Debug("Parser: invalid type [{0}]", line); + return (null); + } + double value; + try + { + value = Convert.ToDouble(valstr); + } + catch (Exception) + { + Logger.Debug("Parser: invalid value [{0}]", line); + return (null); + } + + double rate = 0; + try + { + if (!String.IsNullOrEmpty(ratestr)) + rate = Convert.ToDouble(ratestr); + } + catch (Exception) + { + Logger.Debug("Parser: invalid rate [{0}]", line); + return (null); + } + + if (!string.IsNullOrEmpty(ratestr) && (rate <= 0 || rate > 1)) + { + Logger.Debug("Parser: invalid rate range [{0}]", line); + return (null); + } + + if (!string.IsNullOrEmpty(ratestr) && type != StatsdMetric.StatsdType.StatsdCounter) + { + Logger.Debug("Parser: rate is supported only for Counters [{0}]", line); + return (null); + } + + if (!string.IsNullOrEmpty(ratestr)) + { + value = value/rate; + } + + var metric = new StatsdMetric(name, type, value); + return (metric); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/StatsdPlugin.cs b/src/CollectdWinService/StatsdPlugin.cs new file mode 100644 index 0000000..950ff36 --- /dev/null +++ b/src/CollectdWinService/StatsdPlugin.cs @@ -0,0 +1,115 @@ +using System; +using System.Collections.Generic; +using System.Configuration; +using System.Linq; +using System.Threading; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal class StatsdPlugin : IMetricsReadPlugin + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private bool _delCounters; + private bool _delGauges, _delSets; + private bool _delTimers; + private float[] _percentiles; + private int _port; + private bool _running; + private StatsdAggregator _statsdAggregator; + private StatsdListener _statsdListener; + private Thread _statsdThread; + private bool _timerCount; + private bool _timerLower; + private bool _timerSum; + private bool _timerUpper; + + public StatsdPlugin() + { + _running = false; + } + + public void Configure() + { + var config = ConfigurationManager.GetSection("CollectdWinConfig") as CollectdWinConfig; + if (config == null) + { + throw new Exception("Cannot get configuration section : CollectdWinConfig"); + } + + _port = config.Statsd.Port; + + _delCounters = config.Statsd.DeleteCache.Counters; + _delTimers = config.Statsd.DeleteCache.Timers; + _delGauges = config.Statsd.DeleteCache.Gauges; + _delSets = config.Statsd.DeleteCache.Sets; + + _timerLower = config.Statsd.Timer.Lower; + _timerUpper = config.Statsd.Timer.Upper; + _timerSum = config.Statsd.Timer.Sum; + _timerCount = config.Statsd.Timer.Count; + _percentiles = + (from CollectdWinConfig.StatsdConfig.PercentileConfig percentileConfig in + config.Statsd.Timer.Percentiles + select percentileConfig.Value).ToArray(); + + _statsdAggregator = new StatsdAggregator(_delCounters, _delTimers, _delGauges, _delSets, _timerLower, + _timerUpper, + _timerSum, _timerCount, _percentiles); + Logger.Info("Statsd plugin configured"); + } + + public void Start() + { + if (_running) + return; + + _statsdListener = new StatsdListener(_port, HandleMessage); + _statsdThread = new Thread(_statsdListener.Start); + _statsdThread.Start(); + + _running = true; + Logger.Info("Statsd plugin started"); + } + + public void Stop() + { + if (!_running) + return; + _statsdListener.Stop(); + _statsdThread.Interrupt(); + + _running = false; + Logger.Info("Statsd plugin stopped"); + } + + public IList Read() + { + return (_statsdAggregator.Read()); + } + + public void HandleMessage(string message) + { + IList metrics = StatsdMetricParser.Parse(message); + foreach (StatsdMetric metric in metrics) + { + _statsdAggregator.AddMetric(metric); + } + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/TypesDB.cs b/src/CollectdWinService/TypesDB.cs new file mode 100644 index 0000000..c3c9b65 --- /dev/null +++ b/src/CollectdWinService/TypesDB.cs @@ -0,0 +1,196 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; +using System.Text.RegularExpressions; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + public enum DsType + { + Absolute, + Counter, + Derive, + Gauge + }; + + public enum Status + { + Success, + Failure + }; + + public class DataSource + { + public DataSource(string name, DsType type, double min, double max) + { + Name = name; + Type = type; + Min = min; + Max = max; + } + + public string Name { get; set; } + public DsType Type { get; set; } + public double Min { get; set; } + public double Max { get; set; } + } + + internal class DataSetCollection + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private static volatile DataSetCollection _instance; + private static readonly object SyncRoot = new Object(); + + private readonly Dictionary> _dataSetMap; + // + + private DataSetCollection() + { + _dataSetMap = new Dictionary>(); + } + + public static DataSetCollection Instance + { + get + { + if (_instance == null) + { + lock (SyncRoot) + { + if (_instance == null) + { + _instance = new DataSetCollection(); + _instance.Load(); + } + } + } + return _instance; + } + } + + private static Status GetDouble(string dstr, out double val) + { + if (dstr == "u" || dstr == "U") + { + val = Double.NaN; + return (Status.Success); + } + try + { + val = Double.Parse(dstr); + } + catch (Exception) + { + val = Double.NaN; + return (Status.Failure); + } + return (Status.Success); + } + + public void Print() + { + var sb = new StringBuilder(); + foreach (var entry in _dataSetMap) + { + sb.Append(string.Format("\n[{0}] ==>", entry.Key)); + foreach (DataSource ds in entry.Value) + { + sb.Append(string.Format(" [{0}:{1}:{2}:{3}]", ds.Name, ds.Type, ds.Min, ds.Max)); + } + } + Console.WriteLine(sb.ToString()); + } + + public void Load() + { + const string dataSetPattern = @"[\s\t]*(\w+)[\s\t]*(.*)$"; + const string dataSourcePattern = @"(\w+):(ABSOLUTE|COUNTER|DERIVE|GAUGE):([+-]?\w+):([+-]?\w+)[,]?\s*"; + + var dataSetRegex = new Regex(dataSetPattern, RegexOptions.IgnoreCase); + var dataSourceRegex = new Regex(dataSourcePattern, RegexOptions.IgnoreCase); + + string fileName = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "types.db"); + string[] lines = File.ReadAllLines(fileName); + + foreach (string line in lines) + { + if (line.StartsWith("#")) + { + continue; + } + Match match = dataSetRegex.Match(line); + if (match.Groups.Count < 3) + { + Logger.Error("types.db: invalid data set [{0}]", line); + continue; + } + string dataSetName = match.Groups[1].Value; + MatchCollection matches = dataSourceRegex.Matches(line); + if (matches.Count < 1) + { + Logger.Error("types.db: invalid data source [{0}]", line); + continue; + } + var dataSourceList = new List(); + foreach (Match m in matches) + { + if (m.Groups.Count != 5) + { + Logger.Error("types.db: cannot parse data source [{0}]", line); + dataSourceList.Clear(); + break; + } + + string dsName = m.Groups[1].Value; + var dstype = (DsType) Enum.Parse(typeof (DsType), m.Groups[2].Value, true); + + double min, max; + + if (GetDouble(m.Groups[3].Value, out min) != Status.Success) + { + Logger.Error("types.db: invalid Min value [{0}]", line); + dataSourceList.Clear(); + break; + } + + if (GetDouble(m.Groups[4].Value, out max) != Status.Success) + { + Logger.Error("types.db: invalid Max value [{0}]", line); + dataSourceList.Clear(); + break; + } + + var ds = new DataSource(dsName, dstype, min, max); + dataSourceList.Add(ds); + } + if (dataSourceList.Count > 0) + { + _dataSetMap[dataSetName] = dataSourceList; + } + } + } + + public IList GetDataSource(string dataSetName) + { + IList dataSourceList; + return (_dataSetMap.TryGetValue(dataSetName, out dataSourceList) ? (dataSourceList) : (null)); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/Util.cs b/src/CollectdWinService/Util.cs new file mode 100644 index 0000000..640b052 --- /dev/null +++ b/src/CollectdWinService/Util.cs @@ -0,0 +1,35 @@ +using System; + +namespace BloombergFLP.CollectdWin +{ + public static class Util + { + public static double GetNow() + { + TimeSpan t = DateTime.UtcNow - new DateTime(1970, 1, 1); + double epoch = t.TotalMilliseconds/1000; + double now = Math.Round(epoch, 3); + return (now); + } + + public static string GetHostName() + { + return (Environment.MachineName.ToLower()); + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/WindowsPerformanceCounterPlugin.cs b/src/CollectdWinService/WindowsPerformanceCounterPlugin.cs new file mode 100644 index 0000000..37a773a --- /dev/null +++ b/src/CollectdWinService/WindowsPerformanceCounterPlugin.cs @@ -0,0 +1,174 @@ +using System; +using System.Collections.Generic; +using System.Configuration; +using System.Diagnostics; +using NLog; + +namespace BloombergFLP.CollectdWin +{ + internal struct Metric + { + public string Category; + public string CollectdPlugin, CollectdPluginInstance, CollectdType, CollectdTypeInstance; + public string CounterName; + public IList Counters; + public string Instance; + public uint ScaleDownFactor; + public uint ScaleUpFactor; + } + + internal class WindowsPerformanceCounterPlugin : IMetricsReadPlugin + { + private static readonly Logger Logger = LogManager.GetCurrentClassLogger(); + private readonly IList _metrics; + private string _hostName; + + public WindowsPerformanceCounterPlugin() + { + _metrics = new List(); + } + + public void Configure() + { + var config = ConfigurationManager.GetSection("CollectdWinConfig") as CollectdWinConfig; + if (config == null) + { + throw new Exception("Cannot get configuration section : CollectdWinConfig"); + } + + _hostName = Util.GetHostName(); + + _metrics.Clear(); + + foreach (CollectdWinConfig.CounterConfig counter in config.WindowsPerformanceCounters.Counters) + { + if (counter.Instance == "*") + { + var cat = new PerformanceCounterCategory(counter.Category); + string[] instances = cat.GetInstanceNames(); + foreach (string instance in instances) + { + // Replace collectd_plugin_instance with the Instance got from counter + AddPerformanceCounter(counter.Category, counter.Name, + instance, counter.ScaleUpFactor, + counter.ScaleDownFactor, counter.CollectdPlugin, + instance, counter.CollectdType, + counter.CollectdTypeInstance); + } + } + else + { + AddPerformanceCounter(counter.Category, counter.Name, + counter.Instance, counter.ScaleUpFactor, + counter.ScaleDownFactor, counter.CollectdPlugin, + counter.CollectdPluginInstance, counter.CollectdType, + counter.CollectdTypeInstance); + } + } + Logger.Info("WindowsPerformanceCounter plugin configured"); + } + + public void Start() + { + Logger.Info("WindowsPerformanceCounter plugin started"); + } + + public void Stop() + { + Logger.Info("WindowsPerformanceCounter plugin stopped"); + } + + public IList Read() + { + var metricValueList = new List(); + foreach (Metric metric in _metrics) + { + var vals = new List(); + foreach (PerformanceCounter ctr in metric.Counters) + { + double val = ctr.NextValue(); + if (metric.ScaleUpFactor > 0) + { + val = val*metric.ScaleUpFactor; + } + else + { + if (metric.ScaleDownFactor > 0) + { + val = val/metric.ScaleDownFactor; + } + } + vals.Add(val); + } + + var metricValue = new MetricValue + { + HostName = _hostName, + PluginName = metric.CollectdPlugin, + PluginInstanceName = metric.CollectdPluginInstance, + TypeName = metric.CollectdType, + TypeInstanceName = metric.CollectdTypeInstance, + Values = vals.ToArray() + }; + + TimeSpan t = DateTime.UtcNow - new DateTime(1970, 1, 1); + double epoch = t.TotalMilliseconds/1000; + metricValue.Epoch = Math.Round(epoch, 3); + + metricValueList.Add(metricValue); + } + return (metricValueList); + } + + private void AddPerformanceCounter(string category, string names, string instance, uint scaleUpFactor, + uint scaleDownFactor, string collectdPlugin, string collectdPluginInstance, string collectdType, + string collectdTypeInstance) + { + string logstr = + string.Format( + "Category:{0} - Instance:{1} - counter:{2} - ScaleUpFactor:{3} - ScaleDownFactor:{4} - CollectdPlugin:{5} - CollectdPluginInstance:{6} - CollectdType:{7} - CollectdTypeInstance:{8}", + category, instance, names, scaleUpFactor, scaleDownFactor, collectdPlugin, collectdPluginInstance, + collectdType, collectdTypeInstance); + + try + { + var metric = new Metric(); + string[] counterList = names.Split(','); + metric.Counters = new List(); + foreach (string ctr in counterList) + metric.Counters.Add(new PerformanceCounter(category, ctr.Trim(), instance)); + metric.Category = category; + metric.Instance = instance; + metric.CounterName = names; + metric.ScaleUpFactor = scaleUpFactor; + metric.ScaleDownFactor = scaleDownFactor; + metric.CollectdPlugin = collectdPlugin; + metric.CollectdPluginInstance = collectdPluginInstance; + metric.CollectdType = collectdType; + metric.CollectdTypeInstance = collectdTypeInstance; + + _metrics.Add(metric); + Logger.Info("Added Performance COUNTER : {0}", logstr); + } + catch (Exception exp) + { + Logger.Error("Got exception : {0}, while adding performance counter: {1}", exp, logstr); + } + } + } +} + +// ---------------------------------------------------------------------------- +// Copyright (C) 2015 Bloomberg Finance L.P. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// ----------------------------- END-OF-FILE ---------------------------------- \ No newline at end of file diff --git a/src/CollectdWinService/app.config b/src/CollectdWinService/app.config new file mode 100644 index 0000000..dc1e42d --- /dev/null +++ b/src/CollectdWinService/app.config @@ -0,0 +1,118 @@ + + + + + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/src/CollectdWinService/types.db b/src/CollectdWinService/types.db new file mode 100644 index 0000000..f8b0926 --- /dev/null +++ b/src/CollectdWinService/types.db @@ -0,0 +1,261 @@ +# +# types.db - Data-set specifications for the system statistics collection daemon collectd +# This file is a copy from following URL: +# https://raw.githubusercontent.com/collectd/collectd/master/src/types.db +# +# The types.db file contains one line for each data-set specification. Each +# line consists of two fields delimited by spaces and/or horizontal tabs. +# The first field defines the name of the data-set, while the second field +# defines a list of data-source specifications, delimited by spaces and, +# optionally, a comma (",") right after each list-entry. +# +# The format of the data-source specification has been inspired by RRDtool's +# data-source specification. Each data-source is defined by a quadruple made +# up of the data-source name, type, minimal and maximal values, delimited by +# colons (":"): ds-name:ds-type:min:max. ds-type may be either ABSOLUTE, +# COUNTER, DERIVE, or GAUGE. min and max define the range of valid values for +# data stored for this data-source. If U is specified for either the min or +# max value, it will be set to unknown, meaning that no range checks will +# happen. See rrdcreate(1) for more details. +# +# +absolute value:ABSOLUTE:0:U +apache_bytes value:DERIVE:0:U +apache_connections value:GAUGE:0:65535 +apache_idle_workers value:GAUGE:0:65535 +apache_requests value:DERIVE:0:U +apache_scoreboard value:GAUGE:0:65535 +ath_nodes value:GAUGE:0:65535 +ath_stat value:DERIVE:0:U +backends value:GAUGE:0:65535 +bitrate value:GAUGE:0:4294967295 +blocked_clients value:GAUGE:0:U +bytes value:GAUGE:0:U +cache_eviction value:DERIVE:0:U +cache_operation value:DERIVE:0:U +cache_ratio value:GAUGE:0:100 +cache_result value:DERIVE:0:U +cache_size value:GAUGE:0:U +ceph_bytes value:GAUGE:U:U +ceph_latency value:GAUGE:U:U +ceph_rate value:DERIVE:0:U +changes_since_last_save value:GAUGE:0:U +charge value:GAUGE:0:U +compression_ratio value:GAUGE:0:2 +compression uncompressed:DERIVE:0:U, compressed:DERIVE:0:U +connections value:DERIVE:0:U +conntrack value:GAUGE:0:4294967295 +contextswitch value:DERIVE:0:U +count value:GAUGE:0:U +counter value:COUNTER:U:U +cpufreq value:GAUGE:0:U +cpu value:DERIVE:0:U +current_connections value:GAUGE:0:U +current_sessions value:GAUGE:0:U +current value:GAUGE:U:U +delay value:GAUGE:-1000000:1000000 +derive value:DERIVE:0:U +df_complex value:GAUGE:0:U +df_inodes value:GAUGE:0:U +df used:GAUGE:0:1125899906842623, free:GAUGE:0:1125899906842623 +disk_latency read:GAUGE:0:U, write:GAUGE:0:U +disk_merged read:DERIVE:0:U, write:DERIVE:0:U +disk_octets read:DERIVE:0:U, write:DERIVE:0:U +disk_ops_complex value:DERIVE:0:U +disk_ops read:DERIVE:0:U, write:DERIVE:0:U +disk_time read:DERIVE:0:U, write:DERIVE:0:U +disk_io_time io_time:DERIVE:0:U, weighted_io_time:DERIVE:0:U +dns_answer value:DERIVE:0:U +dns_notify value:DERIVE:0:U +dns_octets queries:DERIVE:0:U, responses:DERIVE:0:U +dns_opcode value:DERIVE:0:U +dns_qtype_cached value:GAUGE:0:4294967295 +dns_qtype value:DERIVE:0:U +dns_query value:DERIVE:0:U +dns_question value:DERIVE:0:U +dns_rcode value:DERIVE:0:U +dns_reject value:DERIVE:0:U +dns_request value:DERIVE:0:U +dns_resolver value:DERIVE:0:U +dns_response value:DERIVE:0:U +dns_transfer value:DERIVE:0:U +dns_update value:DERIVE:0:U +dns_zops value:DERIVE:0:U +drbd_resource value:DERIVE:0:U +duration seconds:GAUGE:0:U +email_check value:GAUGE:0:U +email_count value:GAUGE:0:U +email_size value:GAUGE:0:U +entropy value:GAUGE:0:4294967295 +expired_keys value:GAUGE:0:U +fanspeed value:GAUGE:0:U +file_size value:GAUGE:0:U +files value:GAUGE:0:U +flow value:GAUGE:0:U +fork_rate value:DERIVE:0:U +frequency_offset value:GAUGE:-1000000:1000000 +frequency value:GAUGE:0:U +fscache_stat value:DERIVE:0:U +gauge value:GAUGE:U:U +hash_collisions value:DERIVE:0:U +http_request_methods value:DERIVE:0:U +http_requests value:DERIVE:0:U +http_response_codes value:DERIVE:0:U +humidity value:GAUGE:0:100 +if_collisions value:DERIVE:0:U +if_dropped rx:DERIVE:0:U, tx:DERIVE:0:U +if_errors rx:DERIVE:0:U, tx:DERIVE:0:U +if_multicast value:DERIVE:0:U +if_octets rx:DERIVE:0:U, tx:DERIVE:0:U +if_packets rx:DERIVE:0:U, tx:DERIVE:0:U +if_rx_errors value:DERIVE:0:U +if_rx_octets value:DERIVE:0:U +if_tx_errors value:DERIVE:0:U +if_tx_octets value:DERIVE:0:U +invocations value:DERIVE:0:U +io_octets rx:DERIVE:0:U, tx:DERIVE:0:U +io_packets rx:DERIVE:0:U, tx:DERIVE:0:U +ipt_bytes value:DERIVE:0:U +ipt_packets value:DERIVE:0:U +irq value:DERIVE:0:U +latency value:GAUGE:0:U +links value:GAUGE:0:U +load shortterm:GAUGE:0:5000, midterm:GAUGE:0:5000, longterm:GAUGE:0:5000 +md_disks value:GAUGE:0:U +memcached_command value:DERIVE:0:U +memcached_connections value:GAUGE:0:U +memcached_items value:GAUGE:0:U +memcached_octets rx:DERIVE:0:U, tx:DERIVE:0:U +memcached_ops value:DERIVE:0:U +memory value:GAUGE:0:281474976710656 +memory_lua value:GAUGE:0:281474976710656 +multimeter value:GAUGE:U:U +mutex_operations value:DERIVE:0:U +mysql_commands value:DERIVE:0:U +mysql_handler value:DERIVE:0:U +mysql_locks value:DERIVE:0:U +mysql_log_position value:DERIVE:0:U +mysql_octets rx:DERIVE:0:U, tx:DERIVE:0:U +mysql_bpool_pages value:GAUGE:0:U +mysql_bpool_bytes value:GAUGE:0:U +mysql_bpool_counters value:DERIVE:0:U +mysql_innodb_data value:DERIVE:0:U +mysql_innodb_dblwr value:DERIVE:0:U +mysql_innodb_log value:DERIVE:0:U +mysql_innodb_pages value:DERIVE:0:U +mysql_innodb_row_lock value:DERIVE:0:U +mysql_innodb_rows value:DERIVE:0:U +mysql_select value:DERIVE:0:U +mysql_sort value:DERIVE:0:U +nfs_procedure value:DERIVE:0:U +nginx_connections value:GAUGE:0:U +nginx_requests value:DERIVE:0:U +node_octets rx:DERIVE:0:U, tx:DERIVE:0:U +node_rssi value:GAUGE:0:255 +node_stat value:DERIVE:0:U +node_tx_rate value:GAUGE:0:127 +objects value:GAUGE:0:U +operations value:DERIVE:0:U +packets value:DERIVE:0:U +pending_operations value:GAUGE:0:U +percent value:GAUGE:0:100.1 +percent_bytes value:GAUGE:0:100.1 +percent_inodes value:GAUGE:0:100.1 +pf_counters value:DERIVE:0:U +pf_limits value:DERIVE:0:U +pf_source value:DERIVE:0:U +pf_states value:GAUGE:0:U +pf_state value:DERIVE:0:U +pg_blks value:DERIVE:0:U +pg_db_size value:GAUGE:0:U +pg_n_tup_c value:DERIVE:0:U +pg_n_tup_g value:GAUGE:0:U +pg_numbackends value:GAUGE:0:U +pg_scan value:DERIVE:0:U +pg_xact value:DERIVE:0:U +ping_droprate value:GAUGE:0:100 +ping_stddev value:GAUGE:0:65535 +ping value:GAUGE:0:65535 +players value:GAUGE:0:1000000 +power value:GAUGE:0:U +pressure value:GAUGE:0:U +protocol_counter value:DERIVE:0:U +ps_code value:GAUGE:0:9223372036854775807 +ps_count processes:GAUGE:0:1000000, threads:GAUGE:0:1000000 +ps_cputime user:DERIVE:0:U, syst:DERIVE:0:U +ps_data value:GAUGE:0:9223372036854775807 +ps_disk_octets read:DERIVE:0:U, write:DERIVE:0:U +ps_disk_ops read:DERIVE:0:U, write:DERIVE:0:U +ps_pagefaults minflt:DERIVE:0:U, majflt:DERIVE:0:U +ps_rss value:GAUGE:0:9223372036854775807 +ps_stacksize value:GAUGE:0:9223372036854775807 +ps_state value:GAUGE:0:65535 +ps_vm value:GAUGE:0:9223372036854775807 +pubsub value:GAUGE:0:U +queue_length value:GAUGE:0:U +records value:GAUGE:0:U +requests value:GAUGE:0:U +response_time value:GAUGE:0:U +response_code value:GAUGE:0:U +route_etx value:GAUGE:0:U +route_metric value:GAUGE:0:U +routes value:GAUGE:0:U +segments value:GAUGE:0:65535 +serial_octets rx:DERIVE:0:U, tx:DERIVE:0:U +signal_noise value:GAUGE:U:0 +signal_power value:GAUGE:U:0 +signal_quality value:GAUGE:0:U +smart_poweron value:GAUGE:0:U +smart_powercycles value:GAUGE:0:U +smart_badsectors value:GAUGE:0:U +smart_temperature value:GAUGE:-300:300 +smart_attribute current:GAUGE:0:255, worst:GAUGE:0:255, threshold:GAUGE:0:255, pretty:GAUGE:0:U +snr value:GAUGE:0:U +spam_check value:GAUGE:0:U +spam_score value:GAUGE:U:U +spl value:GAUGE:U:U +swap_io value:DERIVE:0:U +swap value:GAUGE:0:1099511627776 +tcp_connections value:GAUGE:0:4294967295 +temperature value:GAUGE:U:U +threads value:GAUGE:0:U +time_dispersion value:GAUGE:-1000000:1000000 +timeleft value:GAUGE:0:U +time_offset value:GAUGE:-1000000:1000000 +total_bytes value:DERIVE:0:U +total_connections value:DERIVE:0:U +total_objects value:DERIVE:0:U +total_operations value:DERIVE:0:U +total_requests value:DERIVE:0:U +total_sessions value:DERIVE:0:U +total_threads value:DERIVE:0:U +total_time_in_ms value:DERIVE:0:U +total_values value:DERIVE:0:U +uptime value:GAUGE:0:4294967295 +users value:GAUGE:0:65535 +vcl value:GAUGE:0:65535 +vcpu value:GAUGE:0:U +virt_cpu_total value:DERIVE:0:U +virt_vcpu value:DERIVE:0:U +vmpage_action value:DERIVE:0:U +vmpage_faults minflt:DERIVE:0:U, majflt:DERIVE:0:U +vmpage_io in:DERIVE:0:U, out:DERIVE:0:U +vmpage_number value:GAUGE:0:4294967295 +volatile_changes value:GAUGE:0:U +voltage_threshold value:GAUGE:U:U, threshold:GAUGE:U:U +voltage value:GAUGE:U:U +vs_memory value:GAUGE:0:9223372036854775807 +vs_processes value:GAUGE:0:65535 +vs_threads value:GAUGE:0:65535 + +# +# Legacy types +# (required for the v5 upgrade target) +# +arc_counts demand_data:COUNTER:0:U, demand_metadata:COUNTER:0:U, prefetch_data:COUNTER:0:U, prefetch_metadata:COUNTER:0:U +arc_l2_bytes read:COUNTER:0:U, write:COUNTER:0:U +arc_l2_size value:GAUGE:0:U +arc_ratio value:GAUGE:0:U +arc_size current:GAUGE:0:U, target:GAUGE:0:U, minlimit:GAUGE:0:U, maxlimit:GAUGE:0:U +mysql_qcache hits:COUNTER:0:U, inserts:COUNTER:0:U, not_cached:COUNTER:0:U, lowmem_prunes:COUNTER:0:U, queries_in_cache:GAUGE:0:U +mysql_threads running:GAUGE:0:U, connected:GAUGE:0:U, cached:GAUGE:0:U, created:COUNTER:0:U diff --git a/src/installer/CollectdWin.wixproj b/src/installer/CollectdWin.wixproj new file mode 100644 index 0000000..616b172 --- /dev/null +++ b/src/installer/CollectdWin.wixproj @@ -0,0 +1,45 @@ + + + + Debug + x86 + 3.9 + 93cc9e80-b6f5-4768-998e-889bdf5ab579 + 2.0 + CollectdWin-$(Platform) + Package + $(MSBuildExtensionsPath32)\Microsoft\WiX\v3.x\Wix.targets + $(MSBuildExtensionsPath)\Microsoft\WiX\v3.x\Wix.targets + CollectdWin + + + bin\$(Platform)\$(Configuration)\ + obj\$(Platform)\$(Configuration)\ + Debug + + + bin\$(Platform)\$(Configuration)\ + obj\$(Platform)\$(Configuration)\ + + + Debug + bin\$(Platform)\$(Configuration)\ + obj\$(Platform)\$(Configuration)\ + + + bin\$(Platform)\$(Configuration)\ + obj\$(Platform)\$(Configuration)\ + + + + + + + diff --git a/src/installer/Product.wxs b/src/installer/Product.wxs new file mode 100644 index 0000000..906ca98 --- /dev/null +++ b/src/installer/Product.wxs @@ -0,0 +1,105 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/third-party/NLog.dll b/third-party/NLog.dll new file mode 100644 index 0000000..e6fe774 Binary files /dev/null and b/third-party/NLog.dll differ diff --git a/third-party/RabbitMQ.Client.dll b/third-party/RabbitMQ.Client.dll new file mode 100644 index 0000000..87c20e2 Binary files /dev/null and b/third-party/RabbitMQ.Client.dll differ