If you have been observing mysteriously hanging codes, or trouble logging in on headnodes, please read on!
We started receiving reports for hanging processes, mostly for GPU codes. In addition, users who are using tcsh/csh shell as default had difficulties logging into nodes.
Upon further investigation, we found that a storage problem was affecting file locking mechanism on home directories (where most applications keep the configuration files, regardless of where they run).
This problem was very subtle, as it was impacting only a small number of processes and data operations appeared to be working well otherwise.
We have addressed this issue this morning (9/6, 10am) and you should no longer see hanging codes. Please report any ongoing issues to pace-support@oit.gatech.edu.