Internal Production: Last Updated 2009-11-06

Queue Management Policies, Procedures and Commands Reference

• Overview
• Basic Queue Commands
  • midnight
  • pingo / ognip
• Queue Access
  • High Queue Access
  • Special Queue Access
  • Background Queue Limits
• Allocation Enforcement
  • The set_acls Script
  • Manual Allocation Updates
• Restricting User Access
• Altering Job Ordering in the Queue
• Advanced Reservations
• Standing Reservations
• Dynamic Resources (for license checking)

Overview

ARSC HPC systems use workload management software to control the flow of user jobs. This software allows a user to submit a job into a queue. The job will wait in a queue until sufficient resources (e.g. processors) are available for the job to run.

Basic Queue Commands

Midnight

midnight uses the PBS Pro workload management software to control the flow of work.

(See /etc/pbs.conf for the value of PBS_EXEC) 

Pingo / Ognip

pingo uses the PBS Pro workload management software and Cray ALPS to control the flow of work.

(See /etc/pbs.conf for the value of PBS_EXEC) 

Queue Access

Projects receive an allocation of CPU hours. This allocation allows a project to run jobs in a foreground queue.

Other queues may be configured, but may not be available for general use.

High Queue Access Policy

The "high" queue is available to HPCMP customers that have a need to run jobs at a higher priority (than "standard") for a pre-determined period of time. This queue has limits and priority similar to the challenge queue. Requests for access to the "high" queue should be made through the project S/AAA.

Procedure:

1. S/AAA sends a request for a project to be given access to the "high" queue. If the request is made by the PI or a project member, have them contact the S/AAA.
2. Open an RT ticket. At a minimum the following infomation should be documented to assess the request.
1. The project id
2. The anticipated amount of time the project will be given access to the "high" queue.
3. Special needs for jobs run by the project.
4. Remaining allocation available to the project.
3. Have the request reviewed by the "consult" group lead and/or the "specialist" group lead.
4. If the request is approved, add the project to the access list for the "high" queue. If the project request is rejected, send the S/AAA and PI a message explaining why the request was rejected.
5. At the end of the access period, send a message to the PI and S/AAA to verify the project has completed the necessary work. (It may be useful to set a meeting maker event, or RT due date on the ticket).
6. Minor extensions can be accommodated, however projects shouldn't be given continual access to the "high" queue.
7. When work is completed remove access to the "high" queue.
8. Document the change in the RT ticket.

Special Queue Access Policy

The "special" queue is available to customers that have a need to run jobs that do not fit within the normal walltime or processor count limits. This queue has limits and priority similar to the standard queue. Requests for access to the "special" queue should be sent to the ARSC help desk for approval.

Procedure:

1. The user, PI or S/AAA makes a request to allow work to run in the "special" queue.
2. Open an RT ticket. At a minimum the following infomation should be documented to assess the request.
1. The project id or username
2. The anticipated amount of time the project will be given access to the "special" queue.
3. Special needs for jobs run by the project.
4. Limitations on the number of running jobs, walltime, or CPU counts.
5. Remaining allocation available to the project.
3. Have the request reviewed by the "consult" group lead and/or the "specialist" group lead.
4. If the request is approved, add the project to the access list for the "special" queue with the following commands:
   
   # check existing groups given access to special queue
qmgr -c "print server" | grep special
# add project to special queue
qmgr -c "set queue special acl_groups += project_name"
# double check to confirm group was added correctly
qmgr -c "print server" | grep special

   If the project request is rejected, send the requestor a message explaining why the request was rejected.


5. At the end of the access period, send a message to the requestor to verify the project or user has completed the necessary work. (It may be useful to set a meeting maker event, or RT due date on the ticket).
6. Minor extensions can be accommodated, however projects shouldn't be given continual access to the "special" queue.
7. When work is completed remove project access to the "special" queue with the following commands:
   
   # check existing groups given access to special queue
qmgr -c "print server" | grep special
# remove project from the special queue
qmgr -c "set queue special acl_groups -= project_name"
# double check to confirm group was added correctly
qmgr -c "print server" | grep special

   
8. Document the change in the RT ticket.

Background Queue Policy

The "background" queue is required by HPCMP policy. Jobs running in this queue do not use CPU allocation. This queue has the lowest priority. Consultants may alter the number of CPUs available to this queue to ensure that background work doesn't interfere with foreground work. Note "background" work should be allowed to run to completion even when foreground work is waiting.

Procedure:

• Altering limits on midnight
  # check existing limits
qmgr -c "print queue background_lg"
# raise or lower the limit
qmgr -c "set queue background_lg resources_available.ncpus = 128"
  
  
• Altering limits on pingo/ognip
  # check existing limits
qmgr -c "print queue background"
# raise or lower the limit
qmgr -c "set queue background resources_available.mppwidth = 128"
  
  

Allocation Enforcement

Allocations on midnight and pingo are enforced via the use of PBS acl_groups. These Access Control Lists (ACLs) are updated two times a day via the usermon crontab on midnight1 and pingo1. Updates in the allocation database are propagated at these times.

midnight1 Update Times

pingo1 Update Times

The set_acls Script

The set_acls script runs via the usermon crontab to set the UNIX groups that can access each queue (i.e. PBS queue "acl_groups"). The following files control the operation of the set_acls script:

NOTES:
(1) The set_acls.ini file defines which queues are available to standard work and which queues are available to challenge work. This file also defines which projects have access to the challenge queue. Each challenge project must be manually added to this file to have the proper group_acls set. Additional options are available, see the configuration file for details.

Manual Allocation Updates

Allocation records updates can be propogated to either midnight or pingo with the following command:

midnight1 % /projects/consult/sbin/update_allocations
pingo1 % /projects/consult/sbin/update_allocations

Restricting User Queue Access

It may be necessary to block a particular user from the queues due to policy violations or their negative impact to other user's work on the system. In general, it's best to attempt to communicate with the user via email or telephone, however if you cannot reach the user or if the user fails to act within the requested timeframe, a user should be blocked from the queues.

Procedure:

1. Get approval from either the system S&U project lead, consultant group lead, or specialist group lead.
2. Use the "qmgr" command to list out the current user acls for the queue(s).
   
   # check current acl_user settings.
qmgr -c "print server" | grep acl_user


3. Use the "qmgr" command to remove the user's queue access.
   
   # Add access for all users by default if it's not already set
qmgr -c "set queue standard acl_users = +"
# Remove access for the particular user
qmgr -c "set queue standard acl_users += -username"
# enable acls for the queue(s) if they aren't already enabled.
qmgr -c "set queue standard acl_user_enable=True"

NOTE: ACLs may need to be set on more than one queue in order to keep the user from running any jobs (e.g. background, standard, data, debug)
   
   
4. Contact the user in question via email to notify them their access has been removed. If queue access was removed due to a policy violation, be sure to CC the PI for the project as well as the S/AAA (if applicable). Be sure to include the conditions which need to be met to restore queue access.
5. If queue access is removed due to a policy violation, it should be documented in a RT ticket.
6. Queue access can be restored when the user has complied with the request and/or the system S&U project lead, consultant group lead, or specialist group lead indicates the issue has been resolved.
   
# Restore access for the particular user
qmgr -c "set queue standard acl_users -= -username"


7. Notify the user to indicate access has been restored.
8. Add comments to related tickets (if applicable).

Altering Job Ordering in the Queue

The ordering of jobs within the queue can be altered by changing the value of "jprio" for a job. (The default value for jprio is 0).

midnight% qalter -l jprio=1000 PBS_JOBID

Advanced Reservations

Advanced reservation functionality is available within PBS. This functionality allows nodes to be reserved for a particular user or group at some point in the future. This may be useful for code porting, debugging, and approved daily (i.e. operational) activities such as weather forecasting.

Procedure:

1. A user, project, or advocate requests that one or more reservations be created. Generally we prefer at least a 24 hour lead time on reservations.
2. Determine whether the user and/or project has sufficient allocation available to support the requested reservation activity.
3. Pass the request to the consult group lead, specialist group lead, or assistant director for approval
4. If approved, generate the reservation as described below, otherwise contact the customer to explain why their request was not approved
5. Generate the reservation:
   • Midnight Instructions (for an individual user):
     # generate the reservation
pbs_rsub -l select=32:ncpus=4:node_type=4way -U username -R 200902231200 -D 86400 -I 120
# NOTE be sure the reservation is confirmed before continuing
# The pbs_rsub command will return the reservation id (i.e. reservation queue name)


   • Midnight Instructions (for a group/project):
     # generate the reservation
pbs_rsub -l select=32:ncpus=4:node_type=4way -H '*' -G unix_group -U + -R 200902231200 -D 86400 -I 120
# NOTE be sure the reservation is confirmed before continuing
# The pbs_rsub command will return the reservation id (i.e. reservation queue name)


6. Notify the user with the name of the reservation queue. The queue name is available via the "pbs_rstat" or the "qstat -Q" commands.
   
   Example Email:
   

   
   From: ARSC Help Desk <consult@arsc.edu>
   Date: 	Februrary 14, 2009 4:16:29 PM GMT-08:00
   Subject: Reservation Created on Midnight for 12:00 on 2009-02-23 
   
   Dear User,
   
   Per your request we have created a reservation for 128 CPU on midnight.
   
   Start Time:  2009-02-23 12:00
   Resources:   128 CPUs (32- 4way nodes)
   Queue Name:  R12345
   Duration:    8:00:00
   
   You may submit one or more jobs to this reservation with the following 
   syntax:
   
   % qsub -q R12345 -l select=32:ncpus=4:node_type=4way myjob.pbs
   
   or by adding the following PBS directives to your job script:
   
   #!/bin/bash
   #PBS -q R12345
   #PBS -l select=32:ncpus=4:node_type=4way
   #PBS -j oe
   #PBS -l walltime=8:00:00
   
   cd $PBS_O_WORKDIR
   ...
   ...
   
   We request reservation cancellations be made at least 8 business
   hours before the start of the reservation.  If you have questions
   about the use of this reservation, please let us know.
   
   Thanks,
   Don
   	
   -- 
   ARSC Help Desk
   Email: consult@arsc.edu
   Web: http://www.arsc.edu/support/
   Phone: (907) 450-8602
   Fax: (907) 450-8601
   

Standing Reservations

Standing reservations are a special type of reservation which repeats over a period of time. Requests for standing reservations should follow the same guidelines as standard advanced reservations, however keep in mind that standing reservations will occur multiple times and may use significant resources.

Procedure:

1. A user, project, or advocate requests that one or more reservations be created. Generally we prefer at least a 24 hour lead time on reservations.
2. Determine whether the user and/or project has sufficient allocation available to support the requested reservation activity.
3. Pass the request to the consult group lead, specialist group lead, or assistant director for approval
4. If approved, generate the reservation as described below, otherwise contact the customer to explain why their request was not approved
5. Generate the reservation:
   • Midnight Instructions (for an individual user):
     # set the time zone (required for standing reservations)
export PBS_TZID="America/Anchorage";
# generate the reservation
pbs_rsub -l select=32:ncpus=4:node_type=4way -U username -R 200902231200 -r "FREQ=DAILY;UNTIL=20090323" -D 86400 -I 120
# NOTE be sure the reservation is confirmed before continuing
# The pbs_rsub command will return the reservation id (i.e. reservation queue name)


   • Midnight Instructions (for a group/project):
     # set the time zone (required for standing reservations)
export PBS_TZID="America/Anchorage";
# generate the reservation
pbs_rsub -l select=32:ncpus=4:node_type=4way -H '*' -G unix_group -U + -R 200902231200 -r "FREQ=DAILY;UNTIL=20090323" -D 86400 -I 120
# NOTE be sure the reservation is confirmed before continuing
# The pbs_rsub command will return the reservation id (i.e. reservation queue name)


6. Notify the user with the name of the reservation queue. The queue name is available via the "pbs_rstat" or the "qstat -Q" commands.
   
   Example Email:
   

   
   From: ARSC Help Desk <consult@arsc.edu>
   Date: 	Februrary 14, 2009 4:16:29 PM GMT-08:00
   Subject: Standing Reservation Created on Midnight 
   
   Dear User,
   
   Per your request we have created a reservation for 128 CPU on midnight.
   
   Start Time:  2009-02-23 12:00
   Reoccuring:  DAILY
   End Date:    2009-03-23
   Resources:   128 CPUs (32- 4way nodes)
   Queue Name:  S67890
   Duration:    8:00:00
   
   You may submit one or more jobs to this reservation with the following 
   syntax:
   
   % qsub -q S67890 -l select=32:ncpus=4:node_type=4way myjob.pbs
   
   or by adding the following PBS directives to your job script:
   
   #!/bin/bash
   #PBS -q S67890
   #PBS -l select=32:ncpus=4:node_type=4way
   #PBS -j oe
   #PBS -l walltime=8:00:00
   
   cd $PBS_O_WORKDIR
   ...
   ...
   
   We request reservation cancellations be made at least 8 hours business hours before the 
   start of the reservation.  If you have questions about the use of this reservation,
   please let us know.
   
   Thanks,
   Don
   	
   -- 
   ARSC Help Desk
   Email: consult@arsc.edu
   Web: http://www.arsc.edu/support/
   Phone: (907) 450-8602
   Fax: (907) 450-8601
   

Dynamic Resource for License Checking

Several dynamic resource checks are configured on midnight and pingo. These resources ensure that the appropriate number of licenses will be available for a job at run-time and should be added to PBS scripts to indicate the number of licenses required by a job.

Current Resources

License checking is done via the following

• Dynamic resource checks occur on the PBS server (mpbs1 on midnight or sdb on pingo and ognip)
• abaqus - /usr/local/pkg/license_check/bin/abaqus
• abaqus_acad - /usr/local/pkg/license_check/bin/abaqus_acad
• cobalt - /usr/local/pkg/license_check/bin/cobalt
• fluent - /usr/local/pkg/license_check/bin/fluent
• All resource checks make use of the Shared License Buffer (developed by ARL)
  Sample Use:
• /usr/local/pkg/license_check/bin/lic_buffer_client --app cobalt (returns the number of available cobalt licenses