Realtime Control (how to properly pause)

#1

Hi there,

@Christoph thanks a lot for your great work!

I made some experience using virtual flow. It’s great to use! I read in the documentation about Realtime Control of workflows and pausing. How would one properly pause and resume a workflow? I haven’t found any command for pausing.

Thanks a lot, stay healthy & all the best
Chris

PS I am using SGE

1 Like
#2

OK, I guess I would switch the “Terminating Variables” in the current workflow’s control file.
And if I want to continue all after, I would run tools/cf_continue_all.sh

Would this be a proper way to go and pause/resume?

#3

Hi @_Chris_Secker,

Welcome to the community, and thank you for your positive feedback, we really appreciate it :blush:

Yes, you got it perfectly right, you can do it in the control file in the “Terminating Variables” section. More also here in the documentation:
https://docs.virtual-flow.org/documentation/-LdE8RH9UN4HKpckqkX3/using-virtualflow/pausing-and-continuation

Please note that the workflow checks the controlfile for changes not after every ligand which it processes, but only after the number of ligands which is specified by the parameter ligand_check_interval (default 10 if you have not changed it). This is to reduce the I/O generated by the workflow. The time it takes until the workflow stops indeed depends therefore on this value.

Please let me know if you have more questions in this regard.

Stay healthy and safe too, and I hope you will enjoy using VirtualFlow while yielding good results :slight_smile:

All the best,
Christoph

#4

Also, make sure you edit the file in the workflow/control folder which is used by the running workflow, rather than the template control file (all.ctrl) in the tools/templates folder which is only used to prepare the workflow.

#5

Hi Christoph,

thanks for your replies - yes, makes sense to edit in the workflow files. Btw I like how this structured a lot.

I have one more question: after some testing we want to dock some larger libraries now. Our sysadmin asked me that if I will submit now several thousands (similar) jobs whether I can define them as “array jobs” (our cluster uses SGE).

If I start a larger docking now (on 1000 nodes), I was planning to go with:
./vf_start_jobline.sh 1 1000 templates/template1.sge.sh submit 1

I guess this will not define array jobs but just submit 1000 jobs, right?
At least in the template1.sge.sh by default. The qsub option for an array job would be “-t”.

Could I add e.g. -t 1-100 to the options in the template? I am worried that this might produce problems further down, when new jobs are submitted etc. Maybe I just need to test this with a smaller library.

Thanks a lot & best
Chris

qsub man http://gridscheduler.sourceforge.net/htmlman/htmlman1/qsub.html

#6

Hi Chris,

Thanks for letting me know, I’m glad to hear that :blush:

You are right, VirtualFlow uses full jobs and can use job steps (which are only supported by SLURM, LSF, and Moab/TORQUE/PBS). Job arrays are not supported by VirtualFlow.

Because SGE by default only supports a single CPU per job, you will probably need to submit indeed a 1000 jobs in the way you planned (via ./vf_start_jobline.sh). With other batch systems such as SLURM, one can run jobs with multiple queues/CPUs per job, and multiple nodes per jobs as well (via job steps).

I hope all will work out with your first larger screen with VirtualFlow.

Best,
Christoph