Vf_continue.all.sh failing

VFVS VFVS 1
#1

I am trying to understand why the continue script is failing. To troubleshoot the issue I tried to see the queue*.err files under …/workflow/output-files/queues/. I see the following lines

1/1/queue-1-1-17.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/1-1/1-1-17/input-files/ligands/FE/FECBCC/00385.tar.gz: No such file or directory
1/1/queue-1-1-7.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/1-1/1-1-7/input-files/ligands/FF/FFDBCE/00071.tar.gz: No such file or directory
1/1/queue-1-1-21.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/1-1/1-1-21/input-files/ligands/FE/FECBCD/01148.tar.gz: No such file or directory
1/1/queue-1-1-6.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/1-1/1-1-6/input-files/ligands/FF/FFCCDE/00524.tar.gz: No such file or directory
10/1/queue-10-1-19.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/10-1/10-1-19/input-files/ligands/FF/FFCCDD/00446.tar.gz: No such file or directory
10/1/queue-10-1-15.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/10-1/10-1-15/input-files/ligands/FF/FFCCDD/00302.tar.gz: No such file or directory
10/1/queue-10-1-18.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/10-1/10-1-18/input-files/ligands/FD/FDCCDE/01356.tar.gz: No such file or directory
10/1/queue-10-1-20.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/10-1/10-1-20/input-files/ligands/FC/FCCBCE/00360.tar.gz: No such file or directory
2/1/queue-2-1-5.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/2-1/2-1-5/input-files/ligands/FD/FDCCCD/00295.tar.gz: No such file or directory
2/1/queue-2-1-4.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/2-1/2-1-4/input-files/ligands/FD/FDCCCD/00349.tar.gz: No such file or directory
2/1/queue-2-1-19.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/2-1/2-1-19/input-files/ligands/FC/FCCBBD/00189.tar.gz: No such file or directory
2/1/queue-2-1-16.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/2-1/2-1-16/input-files/ligands/FE/FECBDC/00151.tar.gz: No such file or directory
4/1/queue-4-1-16.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/4-1/4-1-16/input-files/ligands/FF/FFCBDD/00394.tar.gz: No such file or directory
4/1/queue-4-1-14.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/4-1/4-1-14/input-files/ligands/FF/FFCBDD/00875.tar.gz: No such file or directory
4/1/queue-4-1-18.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/4-1/4-1-18/input-files/ligands/FE/FEBBCC/00336.tar.gz: No such file or directory
5/1/queue-5-1-18.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/5-1/5-1-18/input-files/ligands/FD/FDCCDD/00838.tar.gz: No such file or directory
5/1/queue-5-1-9.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/5-1/5-1-9/input-files/ligands/FC/FCDCDF/00019.tar.gz: No such file or directory
5/1/queue-5-1-16.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/5-1/5-1-16/input-files/ligands/FD/FDCCDD/00447.tar.gz: No such file or directory
6/1/queue-6-1-21.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/6-1/6-1-21/input-files/ligands/FD/FDCCCE/01626.tar.gz: No such file or directory
6/1/queue-6-1-15.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/6-1/6-1-15/input-files/ligands/FF/FFCBCC/00306.tar.gz: No such file or directory
6/1/queue-6-1-22.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/6-1/6-1-22/input-files/ligands/FD/FDCCCE/00265.tar.gz: No such file or directory
7/1/queue-7-1-10.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-10/input-files/ligands/FD/FDDBDE/00802.tar.gz: No such file or directory
7/1/queue-7-1-12.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-12/input-files/ligands/FD/FDDBDE/00812.tar.gz: No such file or directory
7/1/queue-7-1-13.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-13/input-files/ligands/FD/FDDBDE/00819.tar.gz: No such file or directory
7/1/queue-7-1-5.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-5/input-files/ligands/FF/FFCBCD/01582.tar.gz: No such file or directory
7/1/queue-7-1-3.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-3/input-files/ligands/FD/FDDBDE/00786.tar.gz: No such file or directory
7/1/queue-7-1-9.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-9/input-files/ligands/FD/FDDBDE/00776.tar.gz: No such file or directory
7/1/queue-7-1-8.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-8/input-files/ligands/FD/FDDBDE/00674.tar.gz: No such file or directory
7/1/queue-7-1-11.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/7-1/7-1-11/input-files/ligands/FD/FDDBDE/00751.tar.gz: No such file or directory
8/1/queue-8-1-10.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/8-1/8-1-10/input-files/ligands/FF/FFCBBD/00514.tar.gz: No such file or directory
8/1/queue-8-1-15.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/8-1/8-1-15/input-files/ligands/FD/FDCCDE/00223.tar.gz: No such file or directory
9/1/queue-9-1-8.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-8/input-files/ligands/FE/FECCDD/00627.tar.gz: No such file or directory
9/1/queue-9-1-6.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-6/input-files/ligands/FE/FECCDD/00617.tar.gz: No such file or directory
9/1/queue-9-1-12.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-12/input-files/ligands/FE/FEBBCD/00123.tar.gz: No such file or directory
9/1/queue-9-1-3.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-3/input-files/ligands/FE/FECCDD/00651.tar.gz: No such file or directory
9/1/queue-9-1-20.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-20/input-files/ligands/FB/FBCCCE/00005.tar.gz: No such file or directory
9/1/queue-9-1-23.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-23/input-files/ligands/FE/FECBCD/00324.tar.gz: No such file or directory
9/1/queue-9-1-17.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-17/input-files/ligands/FF/FFCBDE/00136.tar.gz: No such file or directory
9/1/queue-9-1-7.out.err:gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/9-1/9-1-7/input-files/ligands/FE/FECCDD/00561.tar.gz: No such file or directory

As an example, I checked under ../input-files/ligand-library/FF/FFCBDE.tar and did not find the 00136.tar.gz file.

I am not sure why VFVS was expecting this file to be there when the input-library do not contain it. Any pointer why this is happening will be much appreciated.
Thanks.

#2

Hello dmukhop1,

Can you please list the commands you issued prior to getting this error?

Kind regards,

Sorin

#3

Hi Sorin,

Thanks for your reply.
I used grep -r and sed.

  1. From “tools” dir, I issued cd ../workflow/output-files/queues/
  2. Then I issued grep -r "No such" | sed 's/<username>/XXX/g' > output

If I look into the .err files individually I see the following lines (as an example).

tar: FDCCCE/00265.tar.gz: Not found in archive
tar: Exiting with failure status due to previous errors
gzip: /scratch/users/XXX/tmp/XXX/VFVS/t/6-1/6-1-22/input-files/ligands/FD/FDCCCE/00265.tar.gz: No such file or directory
Error was trapped
Error in bash script one-queue.sh
Error on line 822
Environment variables

Hope this helps.

Best
dmukhop1

#4

Sorry for the ambiguity of my question. Let me be more specific:

Did you use vf_prepare_todolists prior to using vf_continue_all ?

S

#5

No, I did not. Are you supposed to? My jobs terminated after the Slurm time limit, and I went back to “tools” and issued vf_continue.all.sh.

#6

Hi Sorin,

If you have experience using the vf-prepare_todolists while continuing submitted job/s, I will be interested to learn. Currently I can only find the following paragraph

Continuation of Joblines

To continue joblines which were stopped, cancelled, or terminated due to other reasons such as errors or cluster problems, the commands vf_continue_jobline.sh and vf_continue_all.sh are provided. If the joblines were stopped via the stop flags in the control files, the stop flags have to be remove/set to false at first before continuing the joblines (otherwise the jobs will terminate immediate since they check these flags at the beginning).

I am not sure if the vf_prepare_todolist will mess up the todolist leading to resubmission of the jobs already completed.

Thanks for your reply.
Best
dmukhop1