Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc] Documentation for FDB.h #49

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 189 additions & 8 deletions src/fdb5/api/FDB.h
Original file line number Diff line number Diff line change
Expand Up @@ -71,72 +71,253 @@ class FDB {

// -------------- Primary API functions ----------------------------

/** Archive binary data to a FDB.
*
* \param handle eckit::message::Message to data to archive
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types can be omitted from the description. IMO this is just hard to keep in sync with the actual implementation

*/
void archive(eckit::message::Message msg);

/** Archive binary data to a FDB.
*
* Reads messages from the eckit#DataHandle and calls archive() the the
* corresponding messages.
*
* \param handle eckit::DataHandle reference data to archive
*/
void archive(eckit::DataHandle& handle);

/** Archive binary data to a FDB .
*
* Internally creates a DataHandle and calls archive().
*
* \param data Pointer to the binary data to archive
* \param length Size of the data to archive with the given
*/
void archive(const void* data, size_t length);
// warning: not high-perf API - makes sure that all the requested fields are archived and there are no data exceeding the request

/** Archive binary data of a mars request and the corresponding binary data to a FDB.
*
* \warning not high-perf API - makes sure that all the requested fields are archived and there are no data exceeding the request
*
* \param request a mars request
* \param handle a data handle pointing to the data
*/
void archive(const metkit::mars::MarsRequest& request, eckit::DataHandle& handle);

// disclaimer: this is a low-level API. The provided key and the corresponding data are not checked for consistency
// Optional callback function is called upon receiving field location from the store.
/** Archive binary data to a FDB.
*
* \warning this is a low-level API. The provided key and the corresponding data are not checked for consistency
* Optional callback function is called upon receiving field location from the store.
*
* \param key Key used for indexing and archiving the data
* \param data Pointer to the binary data to archive
* \param length Size of the data to archive with the given @p key
*/
void archive(const Key& key, const void* data, size_t length);

/// Flushes all buffers and closes all data handles into a consistent DB state
/// @note always safe to call
/** Flush all buffers and closes all data handles into a consistent DB state
*
* \note always safe to call
*/
void flush();

/** Read binary data from an URI
*
* \param uri eckit uri to the data source
*
* \return DataHandle for reading the requested data from
*/
eckit::DataHandle* read(const eckit::URI& uri);

/** Read binary data from an list of URI
*
* \param vector of uris eckit uris to the data source
Copy link
Contributor

@Ozaq Ozaq Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be \params <name or parameter> i.e. \params uris

* \param sorted bool whether the given uris should be sorted
*
* \return DataHandle for reading the requested data from
*/
eckit::DataHandle* read(const std::vector<eckit::URI>& uris, bool sorted = false);

/** Read binary from a ListIterator
*
* \param uris a list iterator which resembles a set of fields which should be read
* \param sorted bool whether the given uris should be sorted
*
* \return DataHandle for reading the requested data from
*/
eckit::DataHandle* read(ListIterator& it, bool sorted = false);

/** Retrieve data which is specified by a MARS request
*
* \param request MarsRequest which describes the data which should be retrieved
*
* \return DataHandle for reading the requested data from
*/
eckit::DataHandle* retrieve(const metkit::mars::MarsRequest& request);

/**
* \todo Comment me pls.
*
* \param request a mars request which should be inspected
*
* \return ListIterator
*/
ListIterator inspect(const metkit::mars::MarsRequest& request);

/** List data present at the archive and which can be retrieved
*
* \param request FDBToolRequest stating which data should be queried
* \param deduplicate bool whether the returned iterator should ignore duplicates
* \param length Size of the data to archive with the given @p key
*
* \return ListIterator for iterating over the set of found items
*/
ListIterator list(const FDBToolRequest& request, bool deduplicate=false);

/** Dump the structural content of the FDB
*
* In particular, in the TOC formulation, enumerate the different entries
* in the Table of Contents (including INIT and CLEAR entries).
* The dump will include information identifying the data files that are
* referenced, and the "Axes" which describe the maximum possible extent of
* the data that is contained in the database.
*
* \return DumpIterator for iterating over the set of found items
*/
DumpIterator dump(const FDBToolRequest& request, bool simple=false);

/// TODO: Is this function superfluous given the control() function?
// \todo Is this function superfluous given the control() function?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was supposed to be a code comment TODO and not a doc comment TODO

StatusIterator status(const FDBToolRequest& request);

/** Wipe data from the database
*
* Deletes FDB databases and the data therein contained. Uses the passed
* request to identify the database to delete. This is equivalent to a
* UNIX rm command.
* This tool deletes either whole databases, or whole indexes within databases
*
* \param request FDBToolRequest stating which data should be queried
* \param doit flag for committing to the wipe (default is dry-run)
* \param porcelain flag print only a list of files to be deleted / that are deleted
* \param unsafeWipeAll flag for omitting all security checks and force a wipe
*
* \return WipeIterator for iterating over the set of wiped items
*/
WipeIterator wipe(const FDBToolRequest& request, bool doit=false, bool porcelain=false, bool unsafeWipeAll=false);

/** Move content of one FDB database.
*
* This locks the source database, make it possible to create a second
* database in another root, duplicates all data. Source data are not automatically removed.
*
* \param request a fdb tool request for the data which should be move
* \param dest destination uri to which the data should be moved
*
* \return MoveIterator for iterating over the set of found items
*/
MoveIterator move(const FDBToolRequest& request, const eckit::URI& dest);

/** Remove duplicate data from the database
*
* Purge duplicate entries from the database and remove the associated data
* (if the data is owned, not adopted).
* Data in the FDB5 is immutable. It is masked, but not damaged or deleted,
* when it is overwritten with new data using the same key.
* If an index or data file only contains masked data
* (i.e. no data that can be obtained using normal requests),
* then those indexes and data files may be removed.
* If an index refers to data that is not owned by the FDB
* (in particular data which has been adopted from an existing FDB4),
* this data will not be removed.
*
* \param request a fdb tool request for the data which should be purged
* \param doit bool if true the purge is triggered, otherwise a dry-run is executed
* \param porcelain bool for printing only those files which are deleted
*
* \return PurgeIterator for iterating over the set of found items
*/
PurgeIterator purge(const FDBToolRequest& request, bool doit=false, bool porcelain=false);

/** Prints information about FDB databases, aggregating the
* information over all the databases visited into a final summary.
*
* \param request FDB tool request for which the stats should be shown
*
* \return StatsIterator for iterating over the set of found items
*/
StatsIterator stats(const FDBToolRequest& request);


/**
* \todo Comment me pls
* \param request FDB tool request
* \param action control action
* \param identifiers identifiers
*
* \return ControlIterator for iterating over the set of found items
*/
ControlIterator control(const FDBToolRequest& request,
ControlAction action,
ControlIdentifiers identifiers);

/**
* \todo Comment me pls
*
* \param request FDB tool request
* \param level maximum level the axis visitor should respect
*
* \return IndexAxis
*/
IndexAxis axes(const FDBToolRequest& request, int level=3);

/**
* \todo Comment me pls
* \param request FDB tool request
* \param level maximum level the axis visitor should respect
*
* \return AxisIterator
*/
AxesIterator axesIterator(const FDBToolRequest& request, int level=3);

/** Check whether a specific control identifier is enabled
*
* \param controlIdentifier a given control identifier
*
* \return bool true or false, depending on the internal status of the FDB
*/
bool enabled(const ControlIdentifier& controlIdentifier) const;

/** Return whether a flush of the FDB is needed
*
* \return true if an archive has happened and a flush is needed
*/
bool dirty() const;

/** Register an archive callback
*
* \param callback an archive callback which should be triggered during archive
*/
void registerArchiveCallback(ArchiveCallback callback);

/** Register a flush callback
*
* \param callback an flush callback which should be triggered during flushing
*/
void registerFlushCallback(FlushCallback callback);

// -------------- API management ----------------------------

/// ID used for hashing in the Rendezvous hash. Should be unique.
const std::string id() const;

FDBStats stats() const;

FDBStats internalStats() const;

const std::string& name() const;

const Config& config() const;

void disable();

bool disabled() const;

private: // methods
Expand Down
Loading