Generate standard drumset from online instruments spreadsheet

The spreadsheet is the source of truth for all instrument data except:

* Some text (e.g. drum names) in src/engraving/types/typesconv.cpp.

* Playback data for Muse Sounds and MS Basic.

share/instruments/README.md

@@ -1,10 +1,103 @@
-# Instruments list
+# Instruments
 
-The list of instruments is stored in this online spreadsheet:
-https://docs.google.com/spreadsheets/d/1SwqZb8lq5rfv5regPSA10drWjUAoi65EuMoYtG-4k5s/edit#gid=516529997
+## About the online spreadsheet
 
-When changes have been made to this spreadsheet, please run `update_instruments_xml.py`,
-to update the following files: 
-- `instruments.xml` (used by MuseScore, to load the instruments);
-- `instrumentsxml.h` (a fake header file used to generate translatable strings from; 
-  see also `/tools/translations/run_lupdate.sh`).
+Instrument data is stored in this [Google spreadsheet][Instruments], which
+contains multiple worksheets (aka. 'sheets' or 'tabs') for different data:
+
+- [Instruments]
+- [Channels]
+- [Basics]
+- Etc.
+
+[Instruments]: https://docs.google.com/spreadsheets/d/1SwqZb8lq5rfv5regPSA10drWjUAoi65EuMoYtG-4k5s/edit#gid=516529997
+[Channels]: https://docs.google.com/spreadsheets/d/1SwqZb8lq5rfv5regPSA10drWjUAoi65EuMoYtG-4k5s/edit#gid=504647632
+[Basics]: https://docs.google.com/spreadsheets/d/1SwqZb8lq5rfv5regPSA10drWjUAoi65EuMoYtG-4k5s/edit#gid=457195594
+
+Some sheets are hidden and only accessible via the View menu to those with
+permission. The hidden sheets contain mostly third-party data that is unlikely
+to change, such as constants from the General MIDI specification. Hiding these
+sheets is purely a matter of convenience.
+
+## Editing the spreadsheet
+
+Community members can create issues or comment on the visible sheets to request
+changes. Team members can request permission to edit the spreadsheet directly
+if required.
+
+Anybody can make an editable copy of the spreadsheet on their own Google Drive,
+or download it in a variety of formats (Excel, ODS, CSV, etc.).
+
+## Updating local files
+
+After you make changes to any of the spreadsheet sheets, run the Python script
+`update_instruments_xml.py` to update the following files in the repository:
+
+- `instruments.xml` - Used by MuseScore Studio to load most instrument data.
+
+- `instrumentsxml.h` - A fake header file used to generate translatable strings
+  (see also `tools/translations/run_lupdate.sh`).
+
+- `src/engraving/dom/drumset.cpp` - A C++ source file that contains the
+  standard drumset used to initialize MuseScore Studio's MIDI and playback
+  systems at startup, prior to `instruments.xml` being loaded.
+
+Commit the changes to these files and submit them in a PR.
+
+## Using cached data
+
+If you're running the Python script repeatedly (e.g. during development), you
+can use the script's `-c` or `--cached` option in combination with the `-d` or
+`--download` option to avoid downloading the entire spreadsheet each time.
+
+For example, this will download only the [Instruments] and [Channels] sheets,
+while using cached data for all the other sheets:
+
+```Bash
+./update_instruments_xml.py --cached --download Instruments --download Channels
+./update_instruments_xml.py -c -d Instruments -d Channels
+```
+
+See the script's `-h` or `--help` option for more details.
+
+Make sure that you run the script once with no options as a final step before
+submitting a PR. This ensures you're using the latest data from all sheets.
+
+## Ignoring other people's changes
+
+If somebody else has edited the spreadsheet, and their changes are showing in
+the diff after you run the Python script, you can use the `-p` or `--patch`
+option to `git add` to avoid committing their changes.
+
+```Bash
+git add -p .
+git add --patch .
+```
+
+This triggers an [interactive staging session], where Git shows you each
+distinct 'hunk' (or region) of diff and asks whether you want to 'stage' it
+(i.e. add it to the index) ready to be committed.
+
+[interactive staging session]: https://git-scm.com/book/en/v2/Git-Tools-Interactive-Staging#_staging_patches
+
+You can respond with:
+
+- `y` - Yes, stage this hunk (in order to commit it).
+- `n` - No, don't stage this hunk (i.e. exclude it from the commit).
+- `?` - Help, show [more patch options].
+
+[more patch options]: https://git-scm.com/docs/git-add#Documentation/git-add.txt-patch
+
+After responding `y` (yes) to all your hunks and `n` (no) to everyone else's,
+you can go ahead and create the commit with `git commit`. Use `git checkout .`
+if you want to remove the unstaged changes from the working tree.
+
+## Justification
+
+The main advantage of the spreadsheet is the speed and convenience with which
+we can make large-scale changes to many items at once. Important too is the
+ability to quickly look along rows and columns to compare values and check for
+errors or inconsistencies. Formulas and formatting also help with these tasks.
+
+Our reasons for not storing the actual spreadsheet file itself in the
+repository are set out [here](https://github.com/musescore/MuseScore/pull/26082#issuecomment-2590711797).

share/instruments/instruments.xml

@@ -7308,7 +7308,7 @@
             </Instrument>
             <Instrument id="drumset">
                   <family>drums</family>
-                  <!--Drums for the basic drumset are hard-coded in instrument.cpp for some reason.-->
+                  <!--Drums are hard-coded in drumset.cpp to initialize MIDI and playback systems at startup before instruments.xml is loaded.-->
                   <trackName>Large Drum Kit</trackName>
                   <longName>Drum Kit</longName>
                   <shortName>D. Kit</shortName>

share/instruments/update_instruments_xml.py

@@ -34,6 +34,7 @@ def eprint(*args, **kwargs):
 import csv
 import io
 import os
+import re
 import requests
 import sys
 import xml.etree.ElementTree as ET
@@ -54,6 +55,8 @@ def eprint(*args, **kwargs):
     'GM+GS_Percussion':         '1216482735',
 }
 
+standard_drumset_id = 'drumset'
+
 parser = argparse.ArgumentParser(description='Fetch the latest spreadsheet and generate instruments.xml.')
 parser.add_argument('-c', '--cached', action='store_true', help='Use cached version instead of downloading')
 parser.add_argument('-d', '--download', action='append', choices=sheet_ids.keys(), help='Override cached option for a specific sheet')
@@ -293,7 +296,7 @@ def to_list(str):
         to_subelement(el, instrument, 'transpDia', 'transposeDiatonic')
         to_subelement(el, instrument, 'transpChr', 'transposeChromatic')
 
-        if instrument['id'] in drumsets:
+        if instrument['id'] != standard_drumset_id and instrument['id'] in drumsets:
             for drum in drumsets[instrument['id']].values():
                 pitch = drum['pitch']
                 d_el = ET.SubElement(el, 'Drum')
@@ -498,3 +501,68 @@ def get_comment(instrument, nameType: str, hasTrait: bool):
     ordersTree = ET.parse('orders.xml')
     for order in ordersTree.getroot().findall('Order'):
         add_translatable_string(f, 'engraving/scoreorder', order.find('name').text)
+
+def noteheadgroup(tag):
+    if tag == 'altbrevis':
+        return 'HEAD_BREVIS_ALT'
+
+    if re.match(r'^[a-h](-sharp|-flat)?-name$', tag):
+        tag = tag[:-5] # remove '-name' from end of tag
+
+    return 'HEAD_' + tag.upper().replace('-', '_')
+
+# Generate the standard drumset. This must be hard-coded in C++ to ensure it's
+# available at startup when systems are initialized (engraving, playback, MIDI
+# & MusicXML import), which happens prior to instruments.xml being loaded.
+standard_drumset_cpp_path = '../../src/engraving/dom/drumset.cpp'
+gen_code = ''
+
+for drum in drumsets[standard_drumset_id].values():
+    pitch = drum['pitch']
+    head = 'HEAD_' + drum['head'].upper().replace('-', '_')
+    stem = 'DOWN' if drum['stem'] == '2' else 'UP'
+
+    shortcut = drum['shortcut']
+    shortcut = 'Key_' + shortcut if shortcut and shortcut != null else '0'
+
+    gen_code += f"""
+    // {drum['drum']}
+    smDrumset->drum({pitch}) = DrumInstrument(
+        TConv::userName(DrumNum({pitch})),
+        NoteHeadGroup::{noteheadgroup(drum['head'])},
+        /*line*/ {drum['line']},
+        DirectionV::{stem},
+        /*panelRow*/ {drum['row']},
+        /*panelColumn*/ {drum['col']},
+        /*voice*/ {drum['voice']},
+        /*shortcut*/ {shortcut});
+"""
+
+with open(standard_drumset_cpp_path, newline='\n', encoding='utf-8') as file:
+    old_code = file.read()
+
+old_lines = old_code.split('\n')
+begin_line = -1
+end_line = -1
+new_code = ''
+
+for num, line in enumerate(old_lines, 1):
+    if 'BEGIN GENERATED CODE' in line:
+        begin_line = num
+        break
+
+assert begin_line > 0
+
+for num, line in enumerate(old_lines[begin_line:], 1):
+    if 'END GENERATED CODE' in line:
+        end_line = begin_line + num
+        break
+
+assert end_line > begin_line
+
+new_code = '\n'.join(old_lines[:begin_line]) + f'\n{gen_code}\n' + '\n'.join(old_lines[end_line-1:])
+
+if new_code != old_code:
+    eprint('Writing ' + standard_drumset_cpp_path)
+    with open(standard_drumset_cpp_path, 'w', newline='\n', encoding='utf-8') as file:
+        file.write(new_code)