LUCI_CONTEXT is a generic way for LUCI services to pass contextual
information to each other. It has a very simple protocol:
- Application writes a JSON file out (usually a temp file)
- Application sets
LUCI_CONTEXTenvironment variable to point to that file. - Subprocesses are expected to read contents from
LUCI_CONTEXTin whole or part. - If any subprocess needs to add/modify information in the context, it copies
the existing
LUCI_CONTEXTentirely, then makes its modifications on the copy before writing it out to a different file (and updating the envvar appropriately.)
The LUCI_CONTEXT JSON file is always a JSON object (e.g. {...}), and
applications are cooperative in terms of the top-level keys (all known keys for
LUCI_CONTEXT and their meaning should be documented in this file). Every
top-level key also corresponds to a JSON object (never a primitive), to avoid
the temptation to pollute the top-level namespace with multiple
related-but-not-grouped data items.
No implementation should make the assumption that it knows the full set of keys and/or schemas (hence the 'copy-and-modify' portion of the protocol).
Parents should keep any LUCI_CONTEXT files they write out alive for the
subprocess to read them (>= observable lifetime of the subprocess). If
a subprocess intends to outlive its parent, it MUST make its own copy of the
LUCI_CONTEXT file.
Example contents:
{
"local_auth": {
"rpc_port": 10000,
"secret": "aGVsbG8gd29ybGQK",
...
},
"swarming": {
"secret_bytes": "cmFkaWNhbGx5IGNvb2wgc2VjcmV0IHN0dWZmCg=="
},
"luciexe": {
"cache_dir": "/b/s/w/ir/cache"
},
"deadline": {
"soft_deadline": 1600883265.1039423,
"grace_period": 30
}
}There is an easy-to-use library for accessing the contents of LUCI_CONTEXT, as
well as producing new contexts, located
[here][./libs/luci_context/luci_context.py].
For precision, the known keys should be documented with a block of protobuf
which, when encoded in jsonpb, result in the expected values. Implementations
will typically treat LUCI_CONTEXT as pure JSON, but we'd like to make the
implementation more rigorous in the future (hence the strict schema
descriptions). Currently implementing LUCI_CONTEXT in terms of actual
protobufs would be onerous, given the way that this repo is deployed and used.
It's assumed that all of the field names in the proto snippets below EXACTLY correspond to their encoded JSON forms. When encoding in golang, this would be equivalent to specifying the 'OrigName' parameter in the Marshaller.
Local auth specifies where subprocesses can obtain OAuth2 tokens to use when calling other services. It is a reference to a local RPC port, along with some configuration of what this RPC service (called "local auth service") can provide.
message LocalAuth {
message Account {
string id = 1;
string email = 2;
}
int rpc_port = 1;
bytes secret = 2;
repeated Account accounts = 3;
string default_account_id = 4;
}...
The returned tokens MUST have expiration duration longer than 150 sec. Clients of the protocol rely on this.
...
The email may be a special string "-" which means tokens produced by the auth
server are not associated with any particular known email. This may happen when
using tokens that don't have userinfo.email OAuth scope.
...
TODO(vadimsh): Finish this.
This section describes data passed down from the swarming service to scripts running within swarming.
message Swarming {
// The user-supplied secret bytes specified for the task, if any. This can be
// used to pass application or task-specific secret keys, JSON, etc. from the
// task triggerer directly to the task. The bytes will not appear on any
// swarming UI, or be visible to any users of the swarming service.
byte secret_bytes = 1;
}This section describes data passed from a luciexe host (e.g. Buildbucket's
agent in swarming).
message LUCIExe {
// The absolute path of the base cache directory. This directory MAY be on the
// same filesystem as CWD (but is not guaranteed to be). The available caches
// are described in Buildbucket as CacheEntry messages.
string cache_dir = 1;
}
This section describes data passed from LUCI Realms integration.
message Realm {
// Realm name of the task.
// e.g. infra:ci
string name = 1;
}This section describes data passed from ResultDB integrations.
message ResultDB {
string hostname = 1; // e.g. results.api.cr.dev
message Invocation {
string name = 1; // e.g. "invocations/build:1234567890"
string update_token = 2; // required in all mutation requests
}
// The invocation in the current context.
// For example, in a Buildbucket build context, it is the build's invocation.
//
// This is the recommended way to propagate invocation name and update token
// to subprocesses.
Invocation current_invocation = 1;
}This section describes the ResultSink available in the environment.
message ResultSink {
// TCP address (e.g. "localhost:62115") where a ResultSink pRPC server is hosted.
string address = 1;
// secret string required in all ResultSink requests in HTTP header
// `Authorization: ResultSink <auth-token>`
string auth_token = 2;
}The Deadline represents an externally-imposed termination criteria for the
process observing the LUCI_CONTEXT.
Additionally, this contains grace_period which can be used to communicate how
long the external process will allow for clean up once it sends
SIGTERM/Ctrl-Break.
Intermediate applications MUST NOT increase soft_deadline or grace_period.
If the entire Deadline is missing from LUCI_CONTEXT, it should be assumed to
be:
{soft_deadline: infinity, grace_period: 30}
Intermediate applications can 'reserve' portions of soft_deadline and
grace_period by reducing them and then enforcing the reduced times.
*** note
WARNING: Reducing soft_deadline may adversely affect the parent process's
ability to accurately assess if soft_deadline was exceeded. This could affect
reporting indicators such as 'timeout occurred', because the child process may
terminate itself before the parent can send a signal and mark that it has done
so.
Most applications SHOULD only reserve time from grace_period. Those reserving
from soft_deadline should take care to ensure that timeout status will still
be accurately communicated to their parent process, if that's important for
the application.
message Deadline {
// The soft deadline for execution for this context as a 'float' unix
// timestamp (seconds past unix epoch). This is the same as python's
// `time.time()` representation.
//
// If this value is set, processes SHOULD rely on their parent process
// to send SIGTERM/Ctrl-Break at this time.
//
// Parent processes adjusting or setting `soft_deadline` MUST enforce it by
// sending SIGTERM/Ctrl-Break as close to this time as possible, followed
// by SIGKILL/Terminate after `grace_period` additional time.
//
// If `soft_deadline` is 0 consider there to be no stated deadline (i.e.
// infinite).
//
// Processes reading this value can use it to determine if a timeout will
// actually be honored; i.e. if the user asks for 30s to run a process, but
// soft_deadline indicates an end in 10s, your program can react accordingly
// (print warning, adjust user-requested timeout, refuse to run user's
// process, etc.).
//
// Processes can also observe this value in conjunction with
// receiving a signal (i.e. I got a signal after `soft_deadline` then I'm likely
// in a timeout state).
double soft_deadline = 1 [json_name = "soft_deadline"];
// The amount of time (in fractional seconds), processes in this context have
// time to react to a SIGTERM/Ctrl-Break before being SIGKILL/Terminated.
//
// If an intermediate process has a lot of cleanup work to do after its child
// quits (e.g. flushing stats/writing output files/etc.) it SHOULD reduce this
// value for the child process by an appropriate margin.
double grace_period = 2 [json_name = "grace_period"];
}
This section describes data to update a buildbucket build.
message Buildbucket {
// Buildbucket host name.
// E.g. cr-buildbucket.appspot.com.
string hostname = 1 [json_name = "hostname"];
// Token to use to schedule child builds for this build.
string schedule_build_token = 2 [json_name = "schedule_build_token"];
}