Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tools: scripts: Update MAVLink parsing for wiki inclusion #27226

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
+88 −5
Open

Tools: scripts: Update MAVLink parsing for wiki inclusion #27226

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
16fcf00
Tools: scripts: mavlink_parse: add rst support
ES-Alexander Jun 3, 2024
adf87f3
Tools: scripts: mavlink_parse: support development message set
ES-Alexander Jun 3, 2024
ce98da7
Tools: scripts: mavlink_parse: improve robustness
ES-Alexander Jun 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 88 additions & 5 deletions Tools/scripts/mavlink_parse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from dataclasses import dataclass, astuple

from pymavlink.dialects.v20 import (
common, icarous, cubepilot, uAvionix, ardupilotmega
common, icarous, cubepilot, uAvionix, ardupilotmega, development
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be just be 'all'? either that or needs minimal
does this end up with lots of duplicates via the recursion of ardupilot -> common -> minimal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be just be 'all'?

It's nice to break them out into dialects, so that the message definitions can be linked to directly within the MAVLink documentation.

either that or needs minimal

The messages defined in minimal are duplicated in the common doc, so I didn't think it was necessary to break it out further (because the links already work fine).

Happy to make that change if you think it's preferable to link to the minimal docs instead of the common docs for those couple of messages. Thoughts?

does this end up with lots of duplicates via the recursion of ardupilot -> common -> minimal?

Nope - the dialects are checked in order (see determine_dialect), so it just stops when it gets to the first one that matches. The MAVLinkDetector class maps relevant messages to their dialects (and corresponding documentation links) as needed, so the only message duplicates are if the firmware handles the same message in different ways that are checked for (e.g. SYSTEM_TIME can be incoming, (automatically) outgoing, and also requestable, so appears in all three tables).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing minimal.xml and/or parsing of include files and use all.xml?

Check Wiki to see if HEARTBEAT is missing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copying part of my response to Tridge from above:

The messages defined in minimal are duplicated in the common doc, so I didn't think it was necessary to break it out further (because the [documentation] links already work fine).

Happy to make that change if you think it's preferable to link to the minimal docs instead of the common docs for those couple of messages. Thoughts?

... parsing of include files ...

Not sure what you're referring to here - this script imports the dialects from pymavlink; it doesn't know about the underlying xml files, and just checks against which messages pymavlink knows about.

In the case of sub-sets (like common < ardupilotmega), as long as the checks are performed in the correct order the relevant dialect is determined as the first match, and can be used to generate a suitable link to the MAVLink docs. It would be nice to directly query the pymavlink.dialects.v20 import for all the available dialects, but they're not stored in completeness order, and some aren't desired (e.g. test) so manual imports seem necessary at the moment.

)

class MAVLinkDialect(StrEnum):
Expand All @@ -18,6 +18,7 @@ class MAVLinkDialect(StrEnum):
CUBEPILOT = 'cubepilot'
UAVIONIX = 'uAvionix'
ARDUPILOTMEGA = 'ardupilotmega'
DEVELOPMENT = 'development'
UNKNOWN = 'UNKNOWN'


Expand Down Expand Up @@ -152,7 +153,8 @@ class MAVLinkDetector:
ARDUPILOT_URL = 'https://github.com/ArduPilot/ardupilot/tree/{branch}/{source}'
EXPORT_FILETYPES = {
'csv': 'csv',
'markdown': 'md'
'markdown': 'md',
'rst': 'rst',
}

MARKDOWN_INTRO = (
Expand All @@ -170,6 +172,9 @@ class MAVLinkDetector:
' to do something meaningful with it.{unsupported}{stream_groups}'
)

# Convert markdown hyperlinks into rst syntax
RST_INTRO = MARKDOWN_INTRO.replace('[', '`').replace('](', ' <').replace(')', '>`_')

VEHICLES = ('AntennaTracker', 'ArduCopter', 'ArduPlane', 'ArduSub', 'Rover')

def __init__(self, common_files, vehicle='ALL',
Expand All @@ -193,7 +198,12 @@ def __init__(self, common_files, vehicle='ALL',
folder = file.parent.stem
if folder in exclude_libraries:
continue
text = file.read_text()

try:
text = file.read_text()
except FileNotFoundError: # Broken symlink
continue
Comment on lines +204 to +205
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would we ignore this?!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I remember having a reason for it back when I added it, but not exactly what that was. My vague recollection is it was related to an unused ardupilot library import that wasn't linked to / removed properly in one of the vehicle branches (possibly Sub?).

Whatever it was was stopping the parsing from completing, and seemed out of scope for this script to solve, so I added the check and moved on.

If you want more context I can try removing the check and re-running the script for every firmware type and version, to see if I can replicate the original issue?


source = f'{folder}/{file.name}'
if file == self.COMMON_FILE:
for mavlink, ap_message in self.find_requestable_messages(text):
Expand Down Expand Up @@ -262,7 +272,11 @@ def find_named_ints(cls, text: str):
def get_stream_groups(self, vehicle):
stream_groups = ['stream_groups']

text = (self.BASE_DIR / vehicle / self.STREAM_GROUP_FILE).read_text()
try:
text = (self.BASE_DIR / vehicle / self.STREAM_GROUP_FILE).read_text()
except FileNotFoundError: # No stream groups
return []

for group_name, message_data in self.STREAM_GROUPS.findall(text):
stream_groups.extend(sorted(
MAVLinkMessage(self._ap_to_mavlink.get(ap_message, ap_message),
Expand Down Expand Up @@ -422,6 +436,74 @@ def export_markdown(self, file, iterable, branch='master', header=None,

print(name, source, dialect, sep=' | ', file=file)

def export_rst(self, file, iterable, branch='master', header=None,
use_intro=True, **extra_kwargs):
if header == 'ardupilot_wiki':
header = '\n'.join((
'.. _mavlink_support:',
'',
'===============',
'MAVLink Support',
'===============',
'\n',
))

if header:
print(header, file=file)

if use_intro:
commands = stream_groups = unsupported = ''
if extra_kwargs['include_commands']:
commands = ' (and commands)'
if extra_kwargs['include_unsupported']:
unsupported = (
'\n\nKnown :ref:`unsupported messages <mavlink_missing_messages>`'
f'{commands} are shown at the end.'
)
if extra_kwargs['include_stream_groups']:
stream_groups = (
'\n\nThe autopilot includes a set of :ref:`mavlink_stream_groups`'
' for convenience, which allow configuring the stream rates of'
' groups of requestable messages by setting parameter values. '
'It is also possible to manually request messages, and request'
' individual messages be streamed at a specified rate.'
)
vehicle = self.vehicle.replace('ALL', 'ArduPilot')

print(self.RST_INTRO.format(
vehicle=vehicle, commands=commands,
stream_groups=stream_groups, unsupported=unsupported
), '\n', file=file)

for data in iterable:
match data:
case str() as type_:
reference = f'mavlink_{type_}'
heading = type_.title().replace('_', ' ')
source_header = (
'Code Source' if type_ != 'stream_groups' else
'Stream Group Parameter'
)
print(f'\n.. _{reference}:\n\n{heading}\n{"="*len(heading)}\n',
self.get_description(type_).replace('`','``'),
'\n.. csv-table::',
f' :header: MAVLink Message, {source_header}, MAVLink Dialect\n\n',
sep='\n', file=file)
case MAVLinkMessage() as message:
name, source, dialect = message.as_tuple()
if dialect != MAVLinkDialect.UNKNOWN:
msg_url = self.MAVLINK_URL.format(dialect=dialect,
message_name=name.split(':')[0])
name = f'`{name} <{msg_url}>`_'
if source != 'UNSUPPORTED' and not source.startswith('SRn'):
folder = source.split('/')[0]
base = 'libraries/' if folder not in self.VEHICLES else ''
code_url = self.ARDUPILOT_URL.format(branch=branch,
source=base+source)
source = f'`{source} <{code_url}>`_'

print(f' {name}', source, dialect, sep=', ', file=file)


if __name__ == '__main__':
from inspect import signature
Expand All @@ -431,6 +513,7 @@ def export_markdown(self, file, iterable, branch='master', header=None,
default_vehicle = detector_init_params['vehicle'].default
vehicle_options = [default_vehicle, *MAVLinkDetector.VEHICLES]
default_exclusions = detector_init_params['exclude_libraries'].default
format_options = [*MAVLinkDetector.EXPORT_FILETYPES, 'none']

parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
parse_opts = parser.add_argument_group('parsing options')
Expand All @@ -449,7 +532,7 @@ def export_markdown(self, file, iterable, branch='master', header=None,
export_opts.add_argument('-q', '--quiet', action='store_true',
help='Disable printout, only export a file.')
export_opts.add_argument('-f', '--format', default='markdown',
choices=['csv', 'markdown', 'none'],
choices=format_options,
help='Desired format for the exported file.')
export_opts.add_argument('-b', '--branch',
help=('The branch to link to in markdown mode.'
Expand Down
Loading