Project

General

Profile

Submit Grid Job » History » Version 21

Timo Eronen, 2016-12-08 12:56

1 1 Timo Eronen
h1. Submit Grid Job
2
3 12 Timo Eronen
*1. Before you can submit Grid jobs you need to prepare your setup.*
4
5
Finish first the steps described here: https://p55cc-redmine.utu.fi/projects/user-s-page/wiki/Prepare_For_Grid_Usage
6
7
*2. Other preparation*
8
9
You use the Grid resources via the ARC (Advanced Resource Connector) middleware, developed by the Nordugrid community.
10
11
There are basically two ways to submit Grid job using ARC:
12
13
# From ANY other computer but Pleione or Titan Login frontend
14
# From Pleione or Titan Login frontend. (i.e. pleione.utu.fi or titan.utu.fi)
15
16
h3. 2.1 Using the Grid from other computers
17
18
Using the Grid from other computers than Pleione or Titan you need to install ARC client software and configure it. Such tasks
19 13 Timo Eronen
are documented here: https://research.csc.fi/fgci-arc-middleware#1.3.2
20 12 Timo Eronen
21
h3. 2.2 Using the Grid from Pleione or Titan frontend
22
23 14 Timo Eronen
The rest of this guide is just short version of this guide: https://research.csc.fi/fgci-using-arc-middleware
24
25 1 Timo Eronen
So you might want to read the comprehensive guide before running the example.
26 15 Timo Eronen
27 19 Timo Eronen
*3. Running self made binaries and FGCI software via runtime environment*
28 1 Timo Eronen
29 19 Timo Eronen
In principle there are two ways to use the Grid resources:
30
- running self made binaries
31 20 Timo Eronen
- running FGCI pre-installed software via runtime environment
32 19 Timo Eronen
33
*4. Example of running a simple self made binary as a Grid job from Titan login node*
34
35 20 Timo Eronen
To run your self made binary as a Grid job you need the following files:
36 16 Timo Eronen
37
- compiled program (the binary)
38
- job description file
39
- batch file
40
- possibly input file(s)
41
42
The c-source of an example program *gtest.c* :
43
44
<pre>
45
#include <stdio.h>
46
47
int main(void) {
48
49
char *line = NULL;
50
size_t size;
51
52
  printf("Hello UTU.\n");
53
54
  while (getline(&line, &size, stdin) != -1)
55
    printf("%s", line);
56
57
  return(0);
58
59
}
60
</pre>
61
62
Compile the source:
63
64
<pre>
65
gcc -Wall -o gtest gtest.c
66
</pre>
67
68
69
The job description file *gtest.xrsl* :
70
71
<pre>
72
&(executable=gtest.sh)
73
(jobname=g_test)
74 1 Timo Eronen
(runtimeenvironment>="ENV/FGCI")
75 16 Timo Eronen
(join="yes")
76
(stdout=std.out)
77
(cpuTime="1 hours")
78 17 Timo Eronen
(count="1")
79 16 Timo Eronen
(memory="1000")
80
(inputfiles=
81
   ("gtest" "" )
82
   ("gtest.txt" "" )
83
)
84
(outputfiles=
85
   ("gtest.tgz" "" )
86
)
87
</pre>
88
89
The batch file *gtest.sh* :
90
91
<pre>
92
#!/bin/sh
93
echo "Running gtest"
94
module load OpenMPI
95
chmod u+x gtest
96
mpirun ./gtest < gtest.txt > gtest.out
97
tar czf gtest.tgz gtest.out
98 1 Timo Eronen
echo "Done"
99 16 Timo Eronen
exit 0
100
</pre>
101
102
103 17 Timo Eronen
Input file for this test run *gtest.txt* :
104 16 Timo Eronen
105 1 Timo Eronen
<pre>
106
107 16 Timo Eronen
Hello FGCI.
108
109
</pre>
110
111 17 Timo Eronen
*Now you are ready to run the job.*
112
113 16 Timo Eronen
Get proxy to run the job:
114
115
<pre>
116
117
$ arcproxy
118
119
Enter pass phrase for private key:
120
121
Your identity: /DC=org/DC=terena/DC=tcs/C=FI/O=Turun yliopisto/CN=Timo Eronen tke@utu.fi
122
Proxy generation succeeded
123
Your proxy is valid until: 2016-12-08 19:46:02
124
</pre>
125
126
Run the job:
127
128
<pre>
129
$ arcsub gtest.xrsl
130
131
Job submitted with jobid: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
132
</pre>
133
134
Query status of the job:
135
136
<pre>
137
$ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
138
139
Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
140
 Name: g_test
141
 State: Finishing
142
143
Status of 1 jobs was queried, 1 jobs returned information
144 1 Timo Eronen
</pre>
145 16 Timo Eronen
146
Job was not yet finished but after awhile:
147
148
<pre>
149
$ arcstat gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
150 17 Timo Eronen
151 16 Timo Eronen
Job: gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
152
 Name: g_test
153
 State: Finished
154
 Exit Code: 0
155
156
Status of 1 jobs was queried, 1 jobs returned information
157
</pre>
158
159
Get the results:
160
161 1 Timo Eronen
<pre>
162 16 Timo Eronen
$ arcget gsiftp://io-grid.fgci.csc.fi:2811/jobs/3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
163
164
Results stored at: 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm
165
Jobs processed: 1, successfully retrieved: 1, successfully cleaned: 1
166
</pre>
167
168 18 Timo Eronen
And then you are ready to view the results:
169 16 Timo Eronen
170
<pre>
171 1 Timo Eronen
$ ls -l 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/
172
total 8
173 16 Timo Eronen
-rw------- 1 tke admin 148 Dec  8 09:14 gtest.tgz
174
-rw------- 1 tke admin  19 Dec  8 09:14 std.out
175
176 17 Timo Eronen
$ cd 3dxMDmPc7Ypn9NOVEmGrhjGmABFKDmABFKDmXGKKDmABFKDmWJfHjm/
177 16 Timo Eronen
178
$ cat std.out
179
Running gtest
180
Done
181 17 Timo Eronen
182
$ tar xf gtest.tgz
183 16 Timo Eronen
184
$ cat gtest.out
185
186
Hello UTU.
187 1 Timo Eronen
188
Hello FGCI.
189
</pre>
190 18 Timo Eronen
191 19 Timo Eronen
*5. Example of running a Grid job via FGCI runtime environments from Titan login node*
192 18 Timo Eronen
193
Comprehensive info:
194
195
https://research.csc.fi/fgci-using-software-through-runtime-environments
196
197 21 Timo Eronen
[an example will be added later]