Some unofficial opinions of how to choose job resources:
If it is mpi that you're using, you're welcome to divide your runs into whatever nodes and ppn that you'd like, provided the product of the two is equal to total number of cores you'd like to use. Keep in mind, not all of our nodes have the high speed networking that mpi gets it's strength from.
1) Memory often is the first thing to consider. While it's not very helpful to say 'memory depends on your job', it's true. An understanding of how your program handles memory is important - and finding optimizations/reducing excess memory usage can not only improve performance, but reduce your resource needs, which gets you through queue waiting times faster. Additionally, the ways your program stores data is important - double format uses twice as much as single format, etc. Keep in mind there's usually a sacrifice in precision when changing formats, and this may or may not matter depending on application.
That said, mem is the memory for a 1 node job, pmem is for jobs where nodes<1, where pmem is a per-core (ppn) assignment. Total memory for the job is ppn*pmem. For a single node job, mem=ppn*pmem. If the memory requirements for your job are high (<160GB) you're likely better off splitting it across multiple nodes.
2) Nodes is largely determined by your memory requirement - if you need (approximately >=160 GB), then your job could spend a lot of time in queues (depending on current usage) waiting for a single node with that much memory to become available. Mpi allows you to distribute your memory requirements for a single job across multiple nodes, getting around the time spent waiting in the queue.
3) PBS -q is the queue. Compute nodes belong to queues, so this determines what computers (and who else is using them) your stuff runs on.
4) PBS -j determines if output and error files are output, PBS -o is the filename. It's a good idea to specify both.
5) In general, it's a good idea to specify modules in the job script manually. This makes it easier to troubleshoot. .pacemodules will automatically start this for every job and login terminal that you start, but you may decide to change this later on, which would affect all of your job scripts. running a 'module purge' and then your 'module load' commands for each pbs script is also a good way to mark which versions of software you used, which is useful for when others try to reproduce your work.
Please keep in mind, all of this is answered in gory detail at our orientation, which is biweekly. This is interactive, and provides a means for you to talk to us directly about applications and problems, which is mutually beneficial. Information about orientations is available at: http://www.pace.gatech.edu/content/orientation
Most (if not all) of this information is available in the slides too: http://www.pace.gatech.edu/orientation/PACE_Orientation.pdf