Implements basic importer for Audacity project files

docs/audacity.md

@@ -0,0 +1,115 @@
+# Audacity
+
+[Audacity](https://www.audacityteam.org/)
+is "free, open source, cross-platform audio software - 
+an easy-to-use, multi-track audio editor and recorder for 
+Windows, macOS, GNU/Linux and other operating systems."
+
+Audacity runs on desktops and laptops, and has a point and click graphical interface and keyboard shortcuts.
+
+Egret can use [Audacity Project](https://manual.audacityteam.org/man/audacity_projects.html) files as input and output.
+
+*As input*: Audacity Project files can contain labels in [label tracks](https://manual.audacityteam.org/man/label_tracks.html).
+The labels can be used as expectations for tests in a test suite.
+
+
+*As output*: Results from running tools and comparing tool output to the test expectations can be saved as Audacity Project files.
+
+
+## Benefits
+
+Audacity may be helpful for [defining test expectations](#usageDefineExpectations) using the graphical interface.
+
+While there are many programs that can visualise audio, 
+Audacity has built-in tools to define areas of interest in the audio.
+The areas of interest are defined by a start time and an end time, and can include a high and low frequency range as well.
+The areas can be defined by clicking and dragging and can be edited in a table layout.
+These areas of interest can be saved to Audacity Project files and used by Egret as test expectations.
+
+Audacity may be helpful when [reviewing results](#usageReviewResults) to 
+look at the visualisation of the audio and labelled areas of interest.
+
+It can be difficult to create reliable audio recognition tools.
+Egret makes it easy to assess the actual output of tools compared to the expected output.
+Output formats such as CSV or JSON allow automated processing to assess the results.
+Sometimes it is useful to manually assess any problems with the actual output not matching expected output.
+Egret results saved as Audacity Project files can be opened in Audacity, along with the audio file.
+The Audacity interface provides a way to manually evaluate test results compared to test expectations.
+
+
+## <a id="usageDefineExpectations"></a> Usage for defining test expectations
+
+Open Audacity  (version 2 or 3) , and 
+[import the audio file](https://manual.audacityteam.org/man/importing_audio.html)
+that contains the sounds to be recognised. 
+
+Define Labels either in Label Tracks using the 
+[label track tools](https://manual.audacityteam.org/man/label_tracks.html) or use the 
+[Label Editor](https://manual.audacityteam.org/man/labels_editor.html).
+
+Save your work as an [Audacity Project](https://manual.audacityteam.org/man/file_menu_save_project.html).
+Either "Save Project" or "Save Project As...". This will save your tracks as an Audacity Project file.
+Depending on the Audacity version, this might:
+
+- create a file with the extension `.aup` and a folder named the same as the Audacity project with the extension `_data` (version 2); or
+- create a file with the extension `.aup3` (version 3).
+
+Egret can import both `.aup` and `.aup3` files.
+
+To import the labels, Egret only needs the `.aup` or `.aup3` file, not the directory.
+The directory contains the audio imported into Audacity. You may want to keep the directory if you want to change the labels.
+
+Make sure you keep the original audio file you imported into Audacity.
+Egret requires this original audio file to run the analysis tools and get results.
+
+The Audacity project file and the audio file must have the same name, and be in the same folder.
+
+The original audio file is found by Egret by finding a file that has the same file "stem" as the Audacity project file.
+The "stem" is the filename before to the last dot `.` (the extension, e.g. `.wav`).
+Egret finds the original audio file by removing the file extension and comparing the "stem" of the audio file and the Audacity project file.
+
+For example, the Audacity project file `/123/abc.aup` or `/123/abc.aup3` might have an original audio file
+at the path `/123/abc.wav` or `/123/abc.mp3`.
+
+
+## <a id="usageReviewResults"></a> Usage for reviewing results
+
+When running Egret, there is an option to save the results as Audacity Project files using the `--audacity` command line argument.
+
+This will create one Audacity Project file for each combination of test suite, tool, and audio file.
+For example, if you have one test suite with two audio files, and two tools, 
+then the output results will be stored in four Audacity Projects.
+
+Egret saves Audacity Project results as `.aup` files, which can be opened by Audacity version 2 or 3.
+
+Open an Audacity Project file. It will contain a number of Label Tracks.
+I won't have any Audio tracks. The audio file that was used by Egret needs to be imported to show the audio track.
+
+Once the audio file is imported, then the Label Tracks can be used to investigate the results from the analysis tools run using Egret.
+
+
+## Use different Audacity Projects for defining expectations and reviewing results
+
+We recommend using separate, dedicated Audacity Projects for defining expectations and reviewing results.
+
+It should be possible to open two or more Audacity Project files at the same time using 
+[File > New](https://manual.audacityteam.org/man/file_menu.html#new).
+
+As you review the results, you can use any insights to make changes in the other Audacity window that contains the test expectations.
+
+
+## Frequently Asked Questions
+
+### Why can't Egret read the audio files saved as part of the Audacity Project file?
+
+An Audacity project file may include zero, one, or more audio files.
+However, the files are stored in Sun AU format, split into smaller files.
+
+The goal is to be able to load expectations and save results to easily view labels against audio files.
+It is tricky to run tools and match test expectations when the original audio is spilt into parts, 
+and in a format that some tools have trouble reading.
+
+Egret is a tool runner, and does not try to convert or combine the audio files.
+
+Instead, there is the convention that one Audacity project file (`.aup` or `.aup3`) references one original audio file.
+The Audacity project file and the audio file must have the same name, and be in the same folder.

src/Egret.Cli/Commands/TestCommand.cs

@@ -43,6 +43,8 @@ public class TestCommandOptions
 
         public bool Csv { get; set; } = false;
 
+        public bool Audacity { get; set; } = false;
+
         public bool Sequential { get; set; } = false;
     }
 

src/Egret.Cli/Egret.Cli.csproj

@@ -24,9 +24,11 @@
 
     <PackageReference Include="Microsoft.Data.Analysis" Version="0.4.0" />
 
-    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="3.1.9" />
-    <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.9" />
-    <PackageReference Include="Microsoft.Extensions.Logging" Version="3.1.9" />
+    <PackageReference Include="Microsoft.Data.Sqlite" Version="5.0.4" />
+
+    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="5.0.1" />
+    <PackageReference Include="Microsoft.Extensions.Hosting" Version="5.0.0" />
+    <PackageReference Include="Microsoft.Extensions.Logging" Version="5.0.0" />
     <PackageReference Include="morelinq" Version="3.3.2" />
     <PackageReference Include="Nito.AsyncEx" Version="5.1.0" />
 

src/Egret.Cli/Formatters/AudacityResultFormatter.cs

@@ -0,0 +1,152 @@
+namespace Egret.Cli.Formatters
+{
+    using Models.Audacity;
+    using Models.Results;
+    using Processing;
+    using Serialization.Audacity;
+    using System;
+    using System.Collections.Generic;
+    using System.IO;
+    using System.Linq;
+    using System.Threading.Tasks;
+
+    /// <summary>
+    /// Result formatter that stores output in Audacity (2.x) project files.
+    /// </summary>
+    public class AudacityResultFormatter : IResultFormatter
+    {
+        private readonly OutputFile outputFile;
+        private readonly AudacitySerializer serializer;
+
+        public AudacityResultFormatter(OutputFile outputFile, AudacitySerializer serializer)
+        {
+            this.outputFile = outputFile;
+            this.serializer = serializer;
+        }
+
+        public ValueTask DisposeAsync()
+        {
+            // no op
+            return ValueTask.CompletedTask;
+        }
+
+        public ValueTask WriteResultsHeader()
+        {
+            // no op
+            return ValueTask.CompletedTask;
+        }
+
+        public ValueTask WriteResult(int index, TestCaseResult result)
+        {
+            (FileInfo currentFile, Project project) = this.BuildProject(index, result);
+            this.serializer.Serialize(currentFile.FullName, project);
+
+            return ValueTask.CompletedTask;
+        }
+
+        public ValueTask WriteResultsFooter(FinalResults finalResults)
+        {
+            // TODO: what to do with the stats?
+            // for now, just ignore the footer data
+            return ValueTask.CompletedTask;
+        }
+
+        private (FileInfo, Project) BuildProject(int index, TestCaseResult result)
+        {
+            var sourceName = Path.GetFileNameWithoutExtension(result.Context.SourceName);
+            var resultFile = outputFile.GetOutputFile(".aup", sourceName);
+
+            // store the outcome and context
+            var outcome = result.Success ? "All expectations passed" : "One or more expectations failed";
+            var tags = new List<Tag>
+            {
+                new("OverallOutcome", outcome),
+                new("ResultIndex", index.ToString()),
+                new(nameof(result.Context.SourceName), result.Context.SourceName),
+                new(nameof(result.Context.SuiteName), result.Context.SuiteName),
+                new(nameof(result.Context.TestName), result.Context.TestName),
+                new(nameof(result.Context.ToolName), result.Context.ToolName),
+                new(nameof(result.Context.ToolVersion), result.Context.ToolVersion),
+                new(nameof(result.Context.CaseTracker), result.Context.CaseTracker.ToString()),
+                new(nameof(result.Context.ExecutionIndex), result.Context.ExecutionIndex.ToString()),
+                new(nameof(result.Context.ExecutionTime), result.Context.ExecutionTime.ToString()),
+            };
+
+            // record any errors
+            tags.AddRange(result.Errors.Select((error, errorIndex) =>
+                new Tag($"Error{errorIndex:D2}", error))
+            );
+
+            // convert results to tracks and labels
+            var eventTruePositives = new List<Label>();
+            var eventFalsePositives = new List<Label>();
+            var eventTrueNegatives = new List<Label>();
+            var eventFalseNegatives = new List<Label>();
+
+            foreach (var expectationResult in result.Results)
+            {
+                // TODO: add event expectation result to TP, FP, TN, FN label list
+                // TODO: add segment-level result as a tag
+
+                var isSuccessful = expectationResult.Successful;
+                var isSegment = expectationResult.IsSegmentResult;
+
+                var subject = expectationResult.Subject;
+                var subjectName = subject.Name ?? String.Empty;
+                var isPositiveAssertion = subject.IsPositiveAssertion;
+
+                var target = expectationResult.Target;
+
+                var truePositive = expectationResult.Contingency == Contingency.TruePositive;
+                var falsePositive = expectationResult.Contingency == Contingency.FalsePositive;
+                var trueNegative = expectationResult.Contingency == Contingency.TrueNegative;
+                var falseNegative = expectationResult.Contingency == Contingency.FalseNegative;
+
+                foreach (var assertion in expectationResult.Assertions)
+                {
+                    switch (assertion)
+                    {
+                        case SuccessfulAssertion successfulAssertion:
+
+                            break;
+                        case ErrorAssertion errorAssertion:
+
+                            break;
+                        case FailedAssertion failedAssertion:
+
+                            break;
+                        default:
+                            throw new NotImplementedException(
+                                $"Unable to process assertion of type '{assertion.GetType()}'.");
+                    }
+                }
+            }
+
+            // create 4 tracks for TP, FP, TN, FN labels
+            var tracks = new List<LabelTrack>
+            {
+                new("EvTP", 1, 100, 0, eventTruePositives.ToArray()),
+                new("EvFP", 1, 100, 0, eventFalsePositives.ToArray()),
+                new("EvTN", 1, 100, 0, eventTrueNegatives.ToArray()),
+                new("EvFN", 1, 100, 0, eventFalseNegatives.ToArray()),
+            };
+
+            // build the audacity project
+            var project = new Project
+            {
+                Tags = tags.ToArray(),
+                Tracks = tracks.ToArray(),
+                ProjectName = $"{sourceName}_data",
+                Version = "1.3.0",
+                AudacityVersion = "2.4.2",
+                Rate = 44100,
+                SnapTo = "off",
+                SelectionFormat = "hh:mm:ss + milliseconds",
+                FrequencyFormat = "Hz",
+                BandwidthFormat = "octaves"
+            };
+
+            return (resultFile, project);
+        }
+    }
+}

src/Egret.Cli/Formatters/MetaFormatter.cs

@@ -37,11 +37,16 @@ public MetaFormatter(ILogger<TestCommand> logger, TestCommandOptions options, IS
                 formatters.Add(provider.GetRequiredService<HtmlResultFormatter>());
             }
 
-            if (logger.PassThrough(options.Csv, "Using HTML result formatter: {yesNo}", LogLevel.Debug))
+            if (logger.PassThrough(options.Csv, "Using CSV result formatter: {yesNo}", LogLevel.Debug))
             {
                 formatters.Add(provider.GetRequiredService<CsvResultFormatter>());
             }
 
+            if (logger.PassThrough(options.Audacity, "Using Audacity result formatter: {yesNo}", LogLevel.Debug))
+            {
+                formatters.Add(provider.GetRequiredService<AudacityResultFormatter>());
+            }
+
             this.formatters = formatters;
         }
 

src/Egret.Cli/Hosting/HostingSetup.cs

@@ -17,9 +17,12 @@
 using YamlDotNet.Serialization.NamingConventions;
 using Egret.Cli.Serialization.Egret;
 using System.IO.Abstractions;
+using Egret.Cli.Serialization.Audacity;
 
 namespace Egret.Cli.Hosting
 {
+    using Serialization.Xml;
+
     public class HostingSetup
     {
         private static readonly string FilterEgretConsoleName = typeof(EgretConsole).FullName;
@@ -64,8 +67,11 @@ private void ConfigureServices(HostBuilderContext context, IServiceCollection se
             services.AddSingleton(_ => UnderscoredNamingConvention.Instance);
             services.AddSingleton<ConfigDeserializer>();
             services.AddSingleton<DefaultJsonSerializer>();
+            services.AddSingleton<DefaultXmlSerializer>();
             services.AddSingleton<LiterateSerializer>();
             services.AddSingleton<AvianzDeserializer>();
+            services.AddSingleton<AudacitySerializer>();
+            services.AddSingleton<Audacity3Serializer>();
 
             services.AddTransient<TempFactory>();
             services.AddTransient<Executor>();
@@ -81,14 +87,17 @@ private void ConfigureServices(HostBuilderContext context, IServiceCollection se
             services.AddSingleton<CsvResultFormatter>();
             services.AddSingleton<HtmlResultFormatter>();
             services.AddSingleton<MetaFormatter>();
+            services.AddSingleton<AudacityResultFormatter>();
 
             services.AddSingleton<SharedImporter>();
             services.AddSingleton<AvianzImporter>();
+            services.AddSingleton<AudacityImporter>();
             services.AddSingleton<EgretImporter>();
             services.AddSingleton((provider) => new ITestCaseImporter[] {
                 provider.GetRequiredService<SharedImporter>(),
                 provider.GetRequiredService<AvianzImporter>(),
                 provider.GetRequiredService<EgretImporter>(),
+                provider.GetRequiredService<AudacityImporter>(),
             });
             services.AddSingleton<TestCaseImporter>();
         }