var_resources

Simulation functions that use the MPIExecutor with dynamic resource assignment. six_hump_camel and helloworld python scripts are used as example applications, but these could be any MPI application.

Each simulation function uses the resources assigned to this worker to set CPU count and, in some functions, specify GPU usage.

GPUs are not used for the six_hump_camel function, but these tests check the assignment is correct. For an example that runs an actual GPU application, see the forces_gpu tutorial under libensemble/tests/scaling_tests/forces/forces_gpu.

See CUDA_variable_resources for an example where the sim function interrogates available resources and sets explicitly.

var_resources.gpu_variable_resources(H, persis_info, sim_specs, libE_info)

Launches an app and automatically assigns GPU resources.

The six_hump_camel app does not run on the GPU, but this test demonstrates how to automatically assign the GPUs given to this worker via the MPIExecutor.

The method used to assign GPUs will be determined by the MPI runner or user-provided configuration (e.g., by setting the platform or platform_specs options or the LIBE_PLATFORM environment variable).

var_resources.gpu_variable_resources_from_gen(H, persis_info, sim_specs, libE_info)

Launches an app and assigns CPU and GPU resources as defined by the gen.

Otherwise similar to gpu_variable_resources.

var_resources.gpu_variable_resources_subenv(H, persis_info, sim_specs, libE_info)

Launches a chain of apps via bash scripts in different sub-processes.

Different MPI runners are specified for each submit. To run without dry_run these MPI runners need to be present. Dry_run is used by default.

Otherwise, this test is similar to gpu_variable_resources.

var_resources.multi_points_with_variable_resources(H, _, sim_specs, libE_info)

Evaluates either helloworld or six hump camel for a collection of points given in H["x"] via the MPI executor, supporting variable sized simulations/resources, as determined by the generator. The term rset refers to a resource set (the minimal set of resources that can be assigned to each worker). It can be anything from a partition of a node to multiple nodes.

Note that this is also an example that is capable of handling multiple points (sim ids) in each call.

See also

test_uniform_sampling_with_variable_resources.py

var_resources.CUDA_variable_resources(H, _, sim_specs, libE_info)

Launches an app setting GPU resources

The standard test apps do not run on GPU, but demonstrates accessing resource information to set CUDA_VISIBLE_DEVICES, and typical run configuration.

For an equivalent function that auto-assigns GPUs using platform detection, see GPU_variable_resources.

var_resources.py
  1"""
  2Simulation functions that use the MPIExecutor with dynamic resource assignment.
  3``six_hump_camel`` and ``helloworld`` python scripts are used as example
  4applications, but these could be any MPI application.
  5
  6Each simulation function uses the resources assigned to this worker to set CPU
  7count and, in some functions, specify GPU usage.
  8
  9GPUs are not used for the six_hump_camel function, but these tests check the
 10assignment is correct. For an example that runs an actual GPU application, see
 11the forces_gpu tutorial under libensemble/tests/scaling_tests/forces/forces_gpu.
 12
 13See CUDA_variable_resources for an example where the sim function
 14interrogates available resources and sets explicitly.
 15
 16"""
 17
 18__all__ = [
 19    "gpu_variable_resources",
 20    "gpu_variable_resources_from_gen",
 21    "gpu_variable_resources_subenv",
 22    "multi_points_with_variable_resources",
 23    "CUDA_variable_resources",
 24]
 25
 26import os
 27
 28import numpy as np
 29
 30from libensemble.message_numbers import TASK_FAILED, UNSET_TAG, WORKER_DONE
 31from libensemble.resources.resources import Resources
 32from libensemble.sim_funcs.six_hump_camel import six_hump_camel_func
 33from libensemble.tools.test_support import check_gpu_setting, check_mpi_runner
 34
 35
 36def gpu_variable_resources(H, persis_info, sim_specs, libE_info):
 37    """Launches an app and automatically assigns GPU resources.
 38
 39    The six_hump_camel app does not run on the GPU, but this test demonstrates
 40    how to automatically assign the GPUs given to this worker via the MPIExecutor.
 41
 42    The method used to assign GPUs will be determined by the MPI runner or
 43    user-provided configuration (e.g., by setting the ``platform`` or
 44    ``platform_specs`` options or the LIBE_PLATFORM environment variable).
 45
 46    """
 47    x = H["x"][0]
 48    H_o = np.zeros(1, dtype=sim_specs["out"])
 49    dry_run = sim_specs["user"].get("dry_run", False)  # logs run lines instead of running
 50    inpt = " ".join(map(str, x))  # Application input
 51
 52    exctr = libE_info["executor"]
 53
 54    # Launch application via system MPI runner, using assigned resources.
 55    task = exctr.submit(
 56        app_name="six_hump_camel",
 57        app_args=inpt,
 58        auto_assign_gpus=True,
 59        match_procs_to_gpus=True,
 60        stdout="out.txt",
 61        stderr="err.txt",
 62        dry_run=dry_run,
 63    )
 64
 65    if not dry_run:
 66        task.wait()  # Wait for run to complete
 67
 68        # Access app output
 69        with open("out.txt") as f:
 70            H_o["f"] = float(f.readline().strip())  # Read just first line
 71
 72    # Asserts GPU set correctly (for known MPI runners)
 73    check_gpu_setting(task, print_setting=True)
 74
 75    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
 76    return H_o, persis_info, calc_status
 77
 78
 79def gpu_variable_resources_from_gen(H, persis_info, sim_specs, libE_info):
 80    """
 81    Launches an app and assigns CPU and GPU resources as defined by the gen.
 82
 83    Otherwise similar to gpu_variable_resources.
 84    """
 85    x = H["x"][0]
 86    H_o = np.zeros(1, dtype=sim_specs["out"])
 87    dry_run = sim_specs["user"].get("dry_run", False)  # logs run lines instead of running
 88    inpt = " ".join(map(str, x))  # Application input
 89
 90    exctr = libE_info["executor"]  # Get Executor
 91
 92    # Launch application via system MPI runner, using assigned resources.
 93    task = exctr.submit(
 94        app_name="six_hump_camel",
 95        app_args=inpt,
 96        stdout="out.txt",
 97        stderr="err.txt",
 98        dry_run=dry_run,
 99    )
100
101    if not dry_run:
102        task.wait()  # Wait for run to complete
103
104        # Access app output
105        with open("out.txt") as f:
106            H_o["f"] = float(f.readline().strip())  # Read just first line
107
108    # Asserts GPU set correctly (for known MPI runners)
109    check_gpu_setting(task, print_setting=True)
110
111    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
112    return H_o, persis_info, calc_status
113
114
115def _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, mpi_runner):
116    """Used to launch each application in a chain"""
117
118    task = exctr.submit(
119        app_name="six_hump_camel",
120        app_args=inpt,
121        auto_assign_gpus=True,
122        match_procs_to_gpus=True,
123        dry_run=dry_run,
124        env_script=env_script_path,
125        mpi_runner_type=mpi_runner,
126    )
127
128    if isinstance(mpi_runner, dict):
129        mpi_runner = mpi_runner["runner_name"]
130
131    check_mpi_runner(task, mpi_runner, print_setting=True)
132    check_gpu_setting(task, print_setting=True)
133
134
135def gpu_variable_resources_subenv(H, persis_info, sim_specs, libE_info):
136    """Launches a chain of apps via bash scripts in different sub-processes.
137
138    Different MPI runners are specified for each submit. To run without dry_run
139    these MPI runners need to be present. Dry_run is used by default.
140
141    Otherwise, this test is similar to ``gpu_variable_resources``.
142
143    """
144    x = H["x"][0]
145    H_o = np.zeros(1, dtype=sim_specs["out"])
146    dry_run = sim_specs["user"].get("dry_run", False)  # logs run lines instead of running
147    env_script_path = sim_specs["user"]["env_script"]  # Script to run in subprocess
148    inpt = " ".join(map(str, x))  # Application input
149
150    exctr = libE_info["executor"]  # Get Executor
151
152    # Launch application via given MPI runner, using assigned resources.
153    _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, "openmpi")
154    _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, "srun")
155
156    mpi_runner_type = {"mpi_runner": "openmpi", "runner_name": "special_mpi"}
157    _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, mpi_runner_type)
158
159    # Now run in current environment.
160    task = exctr.submit(
161        app_name="six_hump_camel",
162        app_args=inpt,
163        auto_assign_gpus=True,
164        match_procs_to_gpus=True,
165        dry_run=dry_run,
166    )
167    check_mpi_runner(task, "mpich", print_setting=True)
168    check_gpu_setting(task, print_setting=True)
169
170    if not dry_run:
171        task.wait()  # Wait for run to complete
172
173        # Access app output
174        with open("out.txt") as f:
175            H_o["f"] = float(f.readline().strip())  # Read just first line
176
177    # Asserts GPU set correctly (for known MPI runners)
178    check_gpu_setting(task, print_setting=True)
179
180    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
181    return H_o, persis_info, calc_status
182
183
184def multi_points_with_variable_resources(H, _, sim_specs, libE_info):
185    """
186    Evaluates either helloworld or six hump camel for a collection of points
187    given in ``H["x"]`` via the MPI executor, supporting variable sized
188    simulations/resources, as determined by the generator. The term `rset`
189    refers to a resource set (the minimal set of resources that can be assigned
190    to each worker). It can be anything from a partition of a node to multiple
191    nodes.
192
193    Note that this is also an example that is capable of handling multiple
194    points (sim ids) in each call.
195
196    .. seealso::
197        `test_uniform_sampling_with_variable_resources.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_uniform_sampling_with_variable_resources.py>`_ # noqa
198    """
199
200    batch = len(H["x"])
201    H_o = np.zeros(batch, dtype=sim_specs["out"])
202    app = sim_specs["user"].get("app", "helloworld")
203    dry_run = sim_specs["user"].get("dry_run", False)  # dry_run only prints run lines in ensemble.log
204    set_cores_by_rsets = True  # If True use rset count to set num procs, else use all available to this worker.
205    core_multiplier = 1  # Only used with set_cores_by_rsets as a multiplier.
206
207    exctr = libE_info["executor"]  # Get Executor
208    task_states = []
209    for i, x in enumerate(H["x"]):
210        nprocs = None  # Will be as if argument is not present
211        if set_cores_by_rsets:
212            resources = Resources.resources.worker_resources
213            nprocs = resources.num_rsets * core_multiplier
214
215        inpt = None  # Will be as if argument is not present
216        if app == "six_hump_camel":
217            inpt = " ".join(map(str, H["x"][i]))
218
219        task = exctr.submit(
220            app_name=app,
221            app_args=inpt,
222            num_procs=nprocs,
223            stdout="out.txt",
224            stderr="err.txt",
225            dry_run=dry_run,
226        )
227
228        if not dry_run:
229            task.wait()  # Wait for run to complete
230
231            # while(not task.finished):
232            #     time.sleep(0.1)
233            #     task.poll()
234
235        task_states.append(task.state)
236
237        if app == "six_hump_camel":
238            # H_o["f"][i] = float(task.read_stdout())  # Reads whole file
239            with open("out.txt") as f:
240                H_o["f"][i] = float(f.readline().strip())  # Read just first line
241        else:
242            # To return something in test
243            H_o["f"][i] = six_hump_camel_func(x)
244
245    calc_status = UNSET_TAG  # Returns to worker
246    if all(t == "FINISHED" for t in task_states):
247        calc_status = WORKER_DONE
248    elif any(t == "FAILED" for t in task_states):
249        calc_status = TASK_FAILED
250
251    return H_o, calc_status
252
253
254def CUDA_variable_resources(H, _, sim_specs, libE_info):
255    """Launches an app setting GPU resources
256
257    The standard test apps do not run on GPU, but demonstrates accessing resource
258    information to set ``CUDA_VISIBLE_DEVICES``, and typical run configuration.
259
260    For an equivalent function that auto-assigns GPUs using platform detection, see
261    GPU_variable_resources.
262    """
263    x = H["x"][0]
264    H_o = np.zeros(1, dtype=sim_specs["out"])
265    dry_run = sim_specs["user"].get("dry_run", False)  # dry_run only prints run lines in ensemble.log
266
267    # Interrogate resources available to this worker
268    resources = Resources.resources.worker_resources
269    slots = resources.slots
270
271    assert resources.matching_slots, f"Error: Cannot set CUDA_VISIBLE_DEVICES when unmatching slots on nodes {slots}"
272
273    num_nodes = resources.local_node_count
274
275    # Set to slots
276    resources.set_env_to_slots("CUDA_VISIBLE_DEVICES")
277    cores_per_node = resources.slot_count
278
279    # Set to detected GPUs
280    # gpus_per_slot = resources.gpus_per_rset_per_node
281    # resources.set_env_to_slots("CUDA_VISIBLE_DEVICES", multiplier=gpus_per_slot)
282    # cores_per_node = resources.slot_count * gpus_per_slot  # One CPU per GPU
283
284    print(
285        f"Worker {libE_info['workerID']}: CUDA_VISIBLE_DEVICES={os.environ['CUDA_VISIBLE_DEVICES']}"
286        f"\tnodes {num_nodes} ppn {cores_per_node}  slots {slots}"
287    )
288
289    # Create application input file
290    inpt = " ".join(map(str, x))
291    exctr = libE_info["executor"]  # Get Executor
292
293    # Launch application via system MPI runner, using assigned resources.
294    task = exctr.submit(
295        app_name="six_hump_camel",
296        app_args=inpt,
297        num_nodes=num_nodes,
298        procs_per_node=cores_per_node,
299        stdout="out.txt",
300        stderr="err.txt",
301        dry_run=dry_run,
302        # extra_args='--gpus-per-task=1'
303    )
304
305    if not dry_run:
306        task.wait()  # Wait for run to complete
307
308        # Access app output
309        with open("out.txt") as f:
310            H_o["f"] = float(f.readline().strip())  # Read just first line
311
312    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
313    return H_o, calc_status