Resolve "Cray system draining flap"

Closes #158

See merge request aig/cobalt!120

A node was getting added to the candidate list twice when looking over
currently running jobs. resulting in a duplicate node and an inflated
count which resulted in an early break.  When this list is later
converted to a set, the duplicates are removed and results in a set
of nodes too small for the job to use for draining, so the job doesn't
get drained for.

Test case added for the drain flap now works correctly now that a check
has been added so a duplicate node doesn't get added.

Lock doesn't need to be held around this state fetch

Closes #178

See merge request aig/cobalt!119

Resolve "schedule_jobs_interval not respected"

Closes #177

See merge request aig/cobalt!118

This is to improve Cobalt responsiveness during state update.  Combined
with the fix for aig/cobalt#177, this should allow for much faster
scheduling cadences as well and should significantly reduce the delay in
a number of commands.

There is an additional fix here where the check for a starting job
during the cleanup update wasn't working.  This was due to a hidden type
mismatch.  This was exposed by both speedups put together.

This allows the site to configure the base sleeptime between automatic
checks on a per-component basis.

This removes the argument and relies on the config file.  Nothing was
setting the option prior to this, and this is a more consistient way to
handle this configuration.

Preventing a doubleing of cycle time if deferring an active reservation.

Closes #176

See merge request aig/cobalt!117

This needed to happen from is_over rather than is_active.  Not sure how
this got reversed.  Fixing a typo.  This actually causes the start time
to be adjusted on a normal cycle.

showres is actually smart enough to show correct times even if the start
time has not been set for the next cycle period on a natural end in the
prior code.

This reverts commit 4da1cffc.

Reset effective GID before SSH'ing to MOM nodes

Closes #173

See merge request aig/cobalt!116

This commit resolves the issues described in issue 173.

Tagging component .out files with a year and month.

Closes #171

See merge request aig/cobalt!114

Resolve "Enhancement: Start time estimate"

Closes #164

See merge request aig/cobalt!111

This gives a more conservative estimate, but prevents a problem where a
large job would appear to be ready to start long before it would
actually be ready.

Adding in reservation location finding tool

Closes #169

See merge request aig/cobalt!112

COBALT_JOBID substitution now properly handled in debuglog (cobaltlog)

Closes #170

See merge request aig/cobalt!113

This adds the tool for finding a reservation location for cluster
systems.  This commit currently only supports cluster systems.

This will provide a list of hosts appropriate for feeding into the -p
flag of setres.  -t corresponds to duration and -s is the same as a
reservation start time.

--attrs won't work on cluster systems (the Cray should use it, though),
 but all the other flags should work.

Manpage pending.

This adds doc, unit tests and the starttime estimate.  This provides
a naive estimate based on machine hours ahead of a job, score-wise,
between the running jobs and currently eligible jobs.  Jobs in hold
do not get estimates, and putting a job in hold removes the estimate.

This also enforces a minimum interval ahead for start times.  This
is adjustable in the cobalt config.

This is a queue-data only estimate.  Cobalt is not prebuliding a
schedudle and the scheduler behavior is not impacted by this.

This will only show up by default with qstat -fl.  This may also
be requested via the header flag.  This is to prevent current user
script breakage.