Skip to content

Roadmap to v1.0

Jump to bottom Edit New page
Daniel Silverman edited this page Oct 8, 2013 · 10 revisions

Refactoring and additional features for a usable "1.0" release. More functionality will come in post-1.0 versions.

High-Level Design Goals

  • Refactor Nepho core to be more object oriented & modular, to encourage outside collaboration
  • Refactor Nepho into a client/server model, to allow asynchronous build monitoring and actions - is this necessary for 1.0?
  • Break orchestration out into providers
    • Build Vagrant provider
    • Build CloudFormation provider
    • Build OpenStack/Rackspace provider - after 1.0
    • Build Google Compute provider - after 1.0
  • Remove all scenarios/patterns/deployments from Nepho core
    • Create a new structure for "cloudlet" bundles
    • Package no more than 3-5 best-of-breed "cloudlets"
    • Create a web-accessible registry/listing of approved cloudlets
    • Allow for pulling in cloudlets from non-standard repos
  • Build cloudlet management into Nepho core, modeled on "puppet module" command
  • Build scenario management (listing, merging YAMLs, etc.) into Nepho core
  • Build interactive prompting for parameters into Nepho core
  • Provide multiple display/output options (pp, json, yaml) - expand this

Refactoring

We want to adopt a more capable CLI framwork, and create a suitable OOP class structure to model the various objects we want to create. We will use Python Cement 2.0 as the basis.

The class structure represent the various textual objects that are assembled into a representation of the desired pattern and configuration, with super classes that are subclassed based on provider and config system.

Client/Server

Currently nepho is completely synchronous, and once a build is started, there's no way to monitor the progress through the tool, or start other processes, without interacting with provider-specific consoles. This is fine with AWS, but we have limited feedback on puppet builds, and if we wanted to use ansible, this would be more difficult. Instead, we want a nanny process to spawn off and monitor/manage the build process to completion.

This nanny process in its simplest form could be a forked process that would communicate back to the command ine client through a simple RPC service, possibly as a message queue. By ensuring that all objects are serializable, this would be straightforward. A further improvement would be for the nanny process to be launched on a remote host, so that longer builds could allow the user to disconnect and reconnect without affecting the build. Lastly, we could have a "foreground" mode that would not fork, but run the same process in the CLI's memory and process space.

Break Orchestration out into providers

Create a clearer seperation of concerns between Nepho components:

  • Nepho Core should manage the scenario (YAML) config file, and pass along options to the provider.
  • Provider should handle all CloudFormation API pieces, and then hand off to the provisioner.
  • Provisioner should handle system bootstrapping.

Note: While provider/provisioner boundaries are fuzzy, we should work to make them more clear. As an initial proposal/discussion point, I suggest that the provider does only the bare minimum cloud-init and then runs a handoff from provider to provisioner, which is always via a boostrap shell script, rather than Puppet management built into the provider. This would provide maximum flexibility for any given application, be it Puppet, Chef, Ansible, Salt, or just basic shell commands.

Create Cloudlet bundles

A cloudlet consists of the following complete set of components in a single git repository

  • A cloudlet info file, in YAML format. Same content as a puppet Modulefile (name, version, source, author, license, summary, description, projett page)
  • Scenario File(s) - One per scenario, minimum of one, named "default". Should the concept of "environments" apply here, or should that be incorporated into the scenario design?
  • CloudFormation Pattern(s), under an "aws" tree for future expandability
  • Provisioner bootstrap file(s) - TBD (one or more shell scripts, scenario chooses one?)
.nepho
└── cloudlets
    └── mediawiki
        ├── cloudlet.yaml
        ├── patterns
        │   └── aws
        │       ├── single-host.cf
        │       └── two-tier-with-vpc.cf
        ├── provisioners
        │   └── bootstrap.sh
        └── scenarios
            └── default.yaml
            └── multi-tenant.yaml

Questions:

  • Should common CF templates be submoduled or subpathed?
  • Likewise for Puppet bootstrap script?

Example Manpage

Simplified cloud orchestration tool for constructing virtual data centers.

Usage: nepho <command> [<args>...]

Configuration commands::
	register      set clould provider credentials

Shortcut commands::
	create        create a new formation from a scenario
	destroy       destroy a running formation
	open          open the public URL endpoint for the formation
	ssh           connect to the formation's public IP or bastion host

Subcommands, use ``nepho help [subcommand]`` to learn more::
	cloudlet      find, download, and manage cloudlets
	scenario      view and manage cloudlet deployment scenarios
	formation     create and manage cloudlet instances

Example Usage Sequence

  • Search for cloudlets
$ nepho cloudlet:search mediawiki
huit/mediawiki - generic mediawiki awesomeness
tfhartmann/mediawiki - like huit but even better
  • Learn about a cloudlet
$ nepho cloudlet:describe huit/mediawiki
Name: huit/mediawiki
Summary: generic mediawiki awesomeness
Description: This is a generic mediawiki module that supports both single-site and multi-site configurations.
Project URL: https://github.com/huit/nepho-mediawiki
Repository: [email protected]:huit/nepho-mediawiki.git
  • Install a cloudlet
$ nepho cloudlet:install huit/mediawiki
Notice: Preparing to install into /Users/zeno/.nepho/cloudlets ...
Notice: Cloning from [email protected]:huit/nepho-mediawiki.git ...
/Users/zeno/.nepho/cloudlets
└── huit-mediawiki (v0.7.2)
  • List installed cloudlets
$ nepho cloudlet:list
/Users/zeno/.nepho/cloudlets
└── huit-mediawiki (v0.7.2)
  • List scenarios in a cloudlet
$ nepho scenario:list mediawiki
default        a single-host, single-tenant setup
multi-tenant   elastic scaling multi-tenant load-balanced service
  • Describe a scenario
$ nepho scenario:describe mediawiki/multi-tenant
Name: multi-tenant
Summary: elastic scaling multi-tenant load-balanced service
Description: Blah blah blah.

------------------------------------------------
Scenario Configuration:
------------------------------------------------
{ lots of YAML output }
------------------------------------------------

Modify these options by editing the scenario file:
/Users/zeno/.nepho/cloudlets/mediawiki/scenarios/multi-tenant.yaml
  • Create a formation from a scenario
$ nepho formation:create mediawiki/multi-tenant --name agmedia
<what should output be here?>
  • List formations
$ nepho formation:list
<what should output be here?>
  • Get info on a formation
$ nepho formation:describe agmedia
<what should output be here?>
  • Open the URL of a formation
$ nepho formation:open agmedia
<what should output be here?>

Complete list of nepho commands/subcommands

  • register - if AWS credentials are missing, present URL to a nepho wiki page about how to sign up for AWS, followed by prompting for region, key id, and secret key
  • cloudlet:list - list installed cloudlets (see puppet module list)
  • cloudlet:search - download/cache register of cloudlets (JSON), search and return results
  • cloudlet:install - download cloudlet from URL in register, or provided URL (git repo) -- also instantiate submodules?
  • cloudlet:uninstall - delete cloudlet files from local disk
  • cloudlet:update - run git pull on specified cloudlet or all of them
  • scenario:list - list all possible scenarios for a given cloudlet (may just be "default") along with the one-line summary of the scenario
  • scenario:describe - list the given scenario, summary, long description, and YAML output
  • formation:list - list all running nepho-tagged cloudformations in the AWS account
  • formation:create - create a new formation from a given cloudlet/scenario combo (nepho formation:create mediawiki/multi-tier or nepho formation:create mediawiki - implies "default"). give some useful processiong output? the stack id to query later? subscribe to sns topic?
  • formation:destroy - destroy a formation by name (if it is unique) or stack id
  • formation:open - open the tier 1 ELB CNAME in a browser (and print it to CLI)
  • formation:ssh - open a SSH connection to the bastion host (or if its public, the host specified as primary/bastion in the scenario) and print the address to CLI
  • formation:describe - query AWS and describe the state of the running formation
Add a custom footer
Add a custom sidebar
Clone this wiki locally