Explore ARCExplore ARC

Introduction to the Flux cluster and batch computing

By |

Overview

This workshop will provide a brief overview of the components of the Flux Cluster. The main body of the workshop will cover the resource manager and scheduler, creating submissions scripts to run jobs and the options available in them, and hands-on experience. By the end of the workshop, every participant should have created a submission script, submitted a job, tracked its progress, and collected its output. Participants will have several working examples from which to build their own submissions scripts in their own home directories.

Prerequisites

Obtain a user account on Flux. If you do not have a Flux user account, click here to go to the account application page at: https://arc-ts.umich.edu/fluxform/

Register for Duo authentication. See below for details.

This course assumes familiarity with the Linux command line as might be got from the CSCAR/ARC-TS workshop Introduction to the Linux Command Line. In particular, participants should understand how files and folders work, be able to create text files using the nano editor, be able to create and remove files and folders, and understand what input and output redirection are and how to use them.

click here to register

Instructor

Bennet Fauber
Scientific Applications Analyst
Advanced Research Computing — Technology Services

Bennet Fauber is a member of the technical team that administers the Flux cluster.

Materials

In-class handouts

Supplementary materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account and be registered for the Duo authentication service. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. Information about registering for Duo is at the ITS Two Factor Authentication web page.

Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.

Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.

Laptop Preparation

The classroom contains Windows computers, which require your uniqname and UMICH password to login, and which have all necessary software preloaded.

If you want to use a laptop for the course, you are welcome to do so. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We will try to help resolve issues after the workshop.

If you want to try to prepare your laptop for use in the class, please see our web page on Preparing your laptop to use Flux. If you have a Windows computer, we recommend that you install PuTTY/WinSCP from the Blue Disc web site. Macintosh computers typically have the minimum requirements installed as part of the system.

More help

Please use our class mailing list hpc101course@umich.edu for questions, comments, or to seek further assistance.

Advanced batch computing on the Flux cluster

By |

Overview

This course will cover some more advanced topics in cluster computing on the U-M Flux Cluster. Topics to be covered include a review of common parallel programming models and basic use of Flux; dependent and array scheduling; advanced troubleshooting and analysis using checkjob, qstat, and other tools; a brief introduction to scientific computing with Python including parallel processing; and parallel debugging and profiling of C and Fortran code, including logging, gdb (line-oriented debugging), ddt (GUI-based debugging) and map (GUI-based profiling) of MPI and OpenMP programs. We will issue you a temporary allocation to use for the course, or you can use your existing Flux allocations, if any.

Prerequisites

Introduction to Batch Computing on Flux or equivalent. This course assumes familiarity with the Linux command line, text editing on Linux, and a basic understanding of Flux including how to submit and track jobs.

click here to register

Instructors

Dr. Charles J Antonelli
Advocacy and Research Support
LSA Information Technology

Charles is a High Performance Computing Consultant in the Advocacy and Research Support Group of LSAIT at the University of Michigan, where he is responsible for high performance computing support and education, and is an advocate to the Departments of History and Communications. Prior to this, he built a parallel data ingestion component of a novel earth science data assimilation system, a secure packet vault, and worked on the No. 5 ESS Switch at Bell Labs in the 80s. He has taught courses in operating systems, distributed file systems, C++ programming, security, and database application design.

Dr. Alexander Gaenko
Technical Services
Advanced Research Computing

Mark Champe
Advocacy and Research Support
LSA Information Technology

Materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account, a Flux allocation (one is created for each workshop), and be enrolled in Duo. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. The allocation allows you to submit those jobs, executing those applications in parallel on the cluster and charging their resource use against the allocation. Duo is required to help authenticate you to the cluster.


Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.


Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.


Duo Authentication

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

Please enroll in Duo before you come to class.

Laptop Preparation

You do not need to bring your own laptop to class. The classroom contains Windows computers, which require your uniqname and UMICH password to login, and that have all necessary software pre-loaded.

If you want to use a laptop for the course, you are welcome to do so:  please see our web page on Preparing your laptop to use Flux. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We cannot stop to debug connection issues with personal or departmental laptops during the class.

More help

Please use our class mailing list hpc201course@umich.edu for questions, comments, or to seek further assistance.

Hadoop and Spark Workshop

By |

Overview

Learn how to process large amounts (up to terabytes) of data using SQL and/or simple programming models available in Python, Scala, and Java. Computers will be provided to follow along with hands-on examples; users can also bring laptops.

Prerequisites

Intro to the Linux Command Line or equivalent. This course assumes familiarity with the Linux command line.

A user account on Flux. If you do not have a Flux user account, click here to go to the account application page at: https://arc-ts.umich.edu/fluxform/

Duo authentication.

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

Hadoop queue membership. If you did not ask to be put on the training Hadoop queue when applying for a Flux user account, send an email to hpc-support@umich.edu asking to be put on the training queue.

click here to register

Instructor

Brock Palen
Associate Director
ARC-TS

Brock has over 10 years of high performance computing and data intensive computing experience in an academic environment. He currently works with the team at ARC-TS to provide HPC, Data Science, storage, and other research computing services to the University. Brock also is the NSF XSEDE projects Campus Champion representing the schools to this and other national computing infrastructures and organizations.

Materials

Course Preparation

In order to participate successfully in the class exercises, you must have a Flux user account, an MToken, and be added to a Hadoop queue. The user account allows you to log in to the cluster, create, compile, and test applications, and transfer data into Hadoop’s filesystem for processing. The Hadoop queue allows you to submit those jobs, executing those applications in parallel on the cluster.

Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.

Hadoop queue

We’ll add you to the training queue so you can run jobs on the cluster during the course. If you already have an existing Hadoop queue, you can use that as well, if you like.

Duo Authentication

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

More help

Please email hpc-support@umich.edu for questions, comments, or to seek further assistance.

Video, slides available: “Advanced Research Computing at Michigan, An Overview,” Brock Palen, ARC-TS

By | General Interest, News

Video (http://myumi.ch/aAG7x) and slides (http://myumi.ch/aV7kz) are now available from Advanced Research Computing – Technology Services (ARC-TS) Associate Director Brock Palen’s presentation “Advanced Research Computing at Michigan, An Overview.”

Palen gave the talk on June 27, 2016, outlining the resources and services available from ARC-TS as well as from off-campus resource providers.

Data Science Skills Series week 1

By |

Fast data processing with Go

We will demonstrate basic Go using several practical examples of data manipulation.  Go (golang.org) is an open source programming language that has many syntax features of high level languages like Python, but is statically typed, compiled, and has native concurrency support.  For these reasons, Go can be extremely efficient at many complex data processing tasks.  

No registration is necessary.

Data Science Skills Series Week 2

By |

Data processing and visualization in R

This workshop will delve into common data processing and exploration techniques, especially as a prelude to visualization.  The main focus will be the dplyr and data.table packages, which enhance and facilitate the sorts of operations that typically arise when dealing with data, including faster I/O and grouped operations.  For visualization, the focus will be on using ggplot2 and other packages that allow for interactivity.

No registration is necessary.

Data Science Skills Series Web Scraping with Python

By |

Web Scraping with Python

We will provide an overview of how to scrape data from html pages and website APIs using Python. For demonstration purposes, we will scrape sports and weather data. This will mostly be accomplished using the Python Requests and Beautiful Soup modules.

No registration is necessary.

On-campus telecast of XSEDE workshop on MPI

By |

U-M is hosting a telecast of a workshop on MPI (message passing interface) presented by XSEDE and the Pittsburgh Supercomputing Center.

This workshop is intended to give C and Fortran programmers a hands-on introduction to MPI programming. Attendees will leave with a working knowledge of how to write scalable codes using MPI – the standard programming tool of scalable parallel computing.

 

Contact: Simon Adorf (csadorf@umich.edu)

Introduction to the Linux Command Line

By |

This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also generically referred to as “the command line”. Topics include: a brief overview of Linux, the Bash shell, navigating the file system, basic commands, shell redirection, permissions, processes, and the command environment. The workshop will also provide a quick introduction to nano a simple text editor that will be used in subsequent workshops to edit files.

click here to register

Instructor

Kenneth Weiss
IT Project Senior Manager
Medical School Information Services (MSIS)

Ken is a High Performance Computing Consultant in the Computational Research Consulting Division of MSIS at the University of Michigan. He works with a team of IT specialists to provide high performance computing support and training for the Medical School. Prior to this, he spent 21 years managing research computing, including an HPC cluster, for Dr. Charles Sing in the Human Genetics Department.

Materials

Course Preparation

You must register at least three full days prior to the event so that we have time to insure you have proper UM credentials for the workshop. This allows enough time for you to get your account adjusted by ITS in case you do not have access to the Linux systems.

Introduction to the Linux Command Line

By |

This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also generically referred to as “the command line”. Topics include: a brief overview of Linux, the Bash shell, navigating the file system, basic commands, shell redirection, permissions, processes, and the command environment. The workshop will also provide a quick introduction to nano a simple text editor that will be used in subsequent workshops to edit files.

click here to register

Instructor

Kenneth Weiss
IT Project Senior Manager
Medical School Information Services (MSIS)

Ken is a High Performance Computing Consultant in the Computational Research Consulting Division of MSIS at the University of Michigan. He works with a team of IT specialists to provide high performance computing support and training for the Medical School. Prior to this, he spent 21 years managing research computing, including an HPC cluster, for Dr. Charles Sing in the Human Genetics Department.

Materials

Course Preparation

You must register at least three full days prior to the event so that we have time to insure you have proper UM credentials for the workshop. This allows enough time for you to get your account adjusted by ITS in case you do not have access to the Linux systems.