https://wiki.nanobiodata.org/api.php?action=feedcontributions&feedformat=atom&user=SysadminRasulev Lab Wiki - User contributions [en]2026-05-14T12:13:42ZUser contributionsMediaWiki 1.38.2https://wiki.nanobiodata.org/index.php?title=Statistical_Machine_Learning&diff=151Statistical Machine Learning2023-03-20T11:14:42Z<p>Sysadmin: Added LLM section</p>
<hr />
<div>=== <span>Overview</span> ===<br />
<div class="outline"><br />
A crash course to enable gentle introduction into the machine learning techniques and its applications into data science. <br />
<br />
Topics Covered: <br />
<br />
* Basic Concepts <br />
* Data Models <br />
* Machine Learning <br />
* ML Techniques <br />
* Software Toolkit <br />
* Continued Learning<br />
<br />
== <span>Basic Concepts</span> ==<br />
<div class="outline"><br />
=== Definitions: ===<br />
<br />
* '''Statistical Machine Learning''' is a set of tools used to model and understand complex data sets <br />
* '''Data Science''' is a set of techniques in computing to support the analysis of data <br />
** Not very useful without some domain knowledge: it is important to ''know your data''. <br />
* Includes analytic techniques: <br />
** descriptive statistics <br />
** data visualization <br />
** statistical machine learning <br />
** neural networks<br />
** actor-environment models <br />
* Also includes computational techniques: <br />
** database administration <br />
** management of information systems <br />
** parallelization <math display="inline">\rightarrow</math> high performance computing<br />
</div><br />
<br />
=== Basic Concepts ===<br />
<div class="outline"><br />
==== Knowing your data ====<br />
Technical definition: <br />
<br />
Let <math display="inline">n</math> represent a number of distinct '''observations''', and let <math display="inline">p</math> represent a number of '''predictors''' Then, our observed data <math display="inline">\textbf{X}</math> is an <math display="inline">n\times p</math> matrix with row observation vectors <math display="inline">\vec{x}_{1..n}</math> and column predictor vectors <math display="inline">\vec{x}_{1..p}</math>. <br />
<br />
In addition, we will also have '''response''' variable(s) <math display="inline">\textbf{Y}</math>, which is a made up of some <math display="inline">n</math>-length vectors. <br />
<br />
So, our combined dataset consists of <math>[(\vec{x}_1,\vec{y}_1),(\vec{x}_2, \vec{y}_2),...,(\vec{x}_n,\vec{y}_n)]</math><br />
<br />
'''Our Mission''': determine relationships between <math display="inline">\textbf{X}</math> and <math display="inline">\textbf{Y}</math> which are mathematically sound, leading to better understanding<br />
<br />
Typically a table has columns as features, rows as entries<br />
<br />
'''Entries''' might be '''numeric''' or '''categorical'''. <br />
<br />
Data sources are either '''Structured''' or '''Unstructured''': <br />
<br />
* Unstructured data will require some transformation. <br />
<br />
Some data may also be '''time series''' taking a sampling of points over time, contributing to a 3-dimensional '''Data Cub'''<br />
<br />
Several techniques can be used to reduce complex data: <br />
<br />
* '''numeric representation''' mapping of categorical information into numbers. <br />
* '''scaling''' redefine a new range for a predictor vector. <br />
* '''normalization''' redefine a predictor by its mean and standard deviation, giving a normal distribution of values. <br />
* '''dimension reduction''' lose fine grain of data, but gain understandability. <br />
* '''feature extraction''' a data mining technique in which we can generate new predictors from known information<br />
<br />
==== Modeling ====<br />
What is a model?<br />
<br />
'''Very well-known model''': Gravity is a functional model between masses, distances, and force. <math display="block">F = G\frac{m_1m_2}{r^2} \rightarrow g = \frac{G M}{r^2} \rightarrow v(t) = v(0) - gt.</math> '''<math display="block">k = Ae^{\frac{E_a}{K_b T}}.</math>Statistics definition''': <br />
<br />
Let <math display="inline">X = (\vec{x}_1, \vec{x}_2, ..., \vec{x}_p)</math> each of length <math display="inline">n</math>, and <math display="inline">Y = (\vec{y})</math> of length <math display="inline">n</math>, then for <math display="inline">X</math> and <math display="inline">Y</math>, there exists a function with a '''systematic''' <math display="inline">f</math> and '''error term''' <math display="inline">\varepsilon</math>: <math display="block">Y = f(X) + \varepsilon</math>Why do we even estimate <math display="inline">f</math> at all? '''Prediction''' or '''Inference'''<br />
<br />
'''Predictive models''' create an estimator <math display="inline">\hat{f}</math> which we can use to estimate <math display="inline">Y</math> using a sample <math display="inline">X</math> from a larger population: <math display="block">\hat{ Y} = \hat{f}(X)</math> With error: <math display="inline">E(Y - \hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \varepsilon</math>. <br />
<br />
'''Inference models''' are primarily interested in how <math display="inline">Y</math> is affected by <math display="inline">X</math>: <br />
<br />
* What predictors associated with response? <br />
* What is the relationship of predictors to response? <br />
* What is the overall nature of relationship between <math display="inline">Y</math> and the predictors.<br />
<br />
==== Signal vs Noise ====<br />
Consider '''precision''' and '''accuracy'''. <br />
<br />
* Both contribute into data set <br />
<br />
'''noise:''' variation in data which detracts from constructing '''information''', as opposed to '''signal'''–data which is representative of a system under study and contains information. <br />
<br />
High signal to noise allows us to minimize '''reducible error''', caused by sampling technique. <br />
<br />
Different than '''irreducible error''', created by factors we are not measuring.<br />
<br />
==== Error and Fit ====<br />
In Modeling In the terms of modeling, precision of a model is referred to '''variance''' and the accuracy of a model its degree of '''bias'''. <br />
<br />
Generally, overly complex models generate high variance, and can '''over-fit''' to input data, making the model useless to new data.<br />
<br />
Generally, '''Mean Square Error''' or MSE, used to determine goodness-of-fit for model calibration: <math display="inline">MSE = \frac{1}{n}\sum^n_{i=1}(y_i - \hat{f}(x_i))^2</math>. <br />
<br />
Error rate used in classification: <math display="inline">\frac{1}{n} \sum^n_{i=1} I(y_i \ne \hat{y}_i)</math>. <br />
<br />
However for reporting, <math display="inline">R^2</math> statistic is more often used, because it gives a value between 0 and 1 useful to determine how much of variance in <math display="inline">Y</math> is explained by variance in <math display="inline">X</math>: <math display="inline">R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2} = \frac{RSS}{TSS}</math><br />
<br />
Example functions relating <math display="inline">X</math> and <math display="inline">Y</math>:<br />
<br />
* A linear function: <math display="inline">Y = \beta_0 + \beta_1 X + \varepsilon</math> <br />
* A polynomial: <math display="inline">Y = \beta_0 + \beta_1 X + \beta_2 X + ... + \varepsilon</math> <br />
* A natural function: <math display="inline">Y = e^{-\alpha_0 X^{\alpha_1}} + \varepsilon</math> <br />
* A logistical function: <math display="inline">p = \frac{ e^{\beta_0 + \beta_1 X} }{1 + e^{\beta_0 + \beta_1 X} }</math> <br />
* A series of nested <code>if</code> statements <br />
* A series of differential equations: <math display="inline">x' = x_n - \bar{x}_n: y' = y_n - \beta_n x_n'</math><br />
<br />
==== Why Machine Learning? ====<br />
Types of Questions: <br />
<br />
* '''Exact Solution is known''' normal coding problems, linear models, and classical statistics. <br />
* '''Exact Solution is unknown, but can be extracted with work''' work with systems experts and domain knowledge to create code. <br />
* '''Exact Solution is known, but not yet conveyable''' ML is useful. <br />
* '''Exact Solution not known by humans''' ML and/or Deep Learning needed<br />
<br />
''Example:'' consider a prediction of temperature: <br />
<br />
* '''Knowledge Based Models''' <br />
** Physics and atmospheric science based model. <br />
** Up to differential equations on chaotic systems. <br />
** As fine granularity of prediction increases, number of factors and density of data quickly becomes too much for most humans to consider <br />
* '''Data Driven Models''' <br />
** Use ML to provide '''iterative''' gain to reduce error <br />
*** Known data→model creation→point toward new factors.<br />
** Uses a split in training and testing data, or sum of error to move toward the correct answer.<br />
** Human researchers more free to find more data, improve prediction, develop theories<br />
<br />
== <span>Techniques</span> ==<br />
<br />
=== Machine Learning in General ===<br />
'''Supervised Learning:''' the model estimates, error verifies <br />
<br />
* If incorrect, needs user input for correction.<br />
* ''Example'': a computer vision system trained to find features in images via user annotated images. <br />
<br />
'''Unsupervised Learning:''' Clustering/Grouping of similar items<br />
<br />
* Need a similarity measure via feature vectors and ability to adjust weights <br />
* ''Example'': Taste prediction algorithms used in web advertising. <br />
<br />
'''Reinforced Learning''' Model estimates a sequence of guesses <br />
<br />
* Correct if and only if the entire sequence or a parameterized output scoring<br />
* Instant feedback but high compute cost <br />
* ''Example'': Game-play in actor-environment model<br />
<br />
=== Types of Problems and Output ===<br />
'''Numeric''' function maps <math display="inline">X</math> to <math display="inline">Y</math> and output is in <math display="inline">\mathbb{R}</math>. <br />
<br />
'''Categorization''' non-orderable sorting <br />
<br />
'''Clustering''' finding principle ways groups differ <br />
<br />
'''Anomaly Detection''' finding data points which are out of the ordinary <br />
<br />
'''Actor Models''' real-time decision making or detailed simulation<br />
<br />
=== Predictive Models ===<br />
Predictive ML models which are also Linear: <br />
<br />
* Utilize a split of training and test data: test-training or <math display="inline">k</math>-fold cross-validation <br />
* use a function mapping of one or more independent variables to the dependent variables, then re-evaluate to reduce error Includes techniques for mixed model reduction<br />
* Reduction of the number of predictors via '''Lasso''', '''Ridge''', and '''Elastic Net''' techniques<br />
<br />
=== Feature-vector based models ===<br />
Nested <code>if</code> statements try to find decision boundaries by distance between independent data and dependent outcome features have weighted probability most information by Bayesian inference<br />
<br />
* '''Decision Trees''' can be used to create decision models for linearly separable data effectively a neural network with one neuron<br />
* '''Random Forest''' utilizes a number of differently-tuned trees trees provide consensus voting-based approach for non-linearly separable data smaller tree depth typically prevents over-fit<br />
<br />
=== Clustering ===<br />
Groups data into cluster such that distance within clusters is small, and between differing groups is large <br />
<br />
Works with any well-defined &quot;distance&quot; function: Euclidean, Hamming, Inner Product, etc. <br />
<br />
'''k-Means Clustering:''' <br />
<br />
* choose <math display="inline">k</math> number of clusters randomly distribute <math display="inline">k</math> points, '''centroids''', into feature space <br />
* divide and classify data by distance to <math display="inline">k</math> centroids <br />
* move centroids based on center of groups repeats until convergence to some epsilon value<br />
* where points no longer move across iterations <br />
* '''Goodness of Fit''' for <math display="inline">N-M</math> possible values of <math display="inline">k</math>, an inflection in overall likelihood ratio given by probability function for set<br />
<br />
'''K-nearest Neighbors:''' <br />
<br />
Creates a probabilistic decision boundary within a feature space between <math display="inline">K</math> centroids<br />
<br />
Unsupervised system to find structures of data works on majority voting system<br />
<br />
'''Gradient Boosting:'''<br />
<br />
System attempts to find the direction and vector of change in a dimensional field, and follow these iteratively to find local extrema. <br />
<br />
Most use some '''Quasi-Newton Method''' for finding extrema for faster centroid convergence. <br />
<br />
'''Support Vector Machine:''' <br />
<br />
Also known as SVM, applies a classifier into high-dimensional data to split points into groups some use a '''kernel trick.'''<br />
<br />
Generates the inner product space of two arbitrary-dimensional numeric matrix spaces, showing the shape of data<br />
<br />
=== Genetic Algorithms ===<br />
Utilize some adversarial scoring method of initially randomized vectors:<br />
<br />
* ’survivors’ become the basis of new models similar iterative concept to gradient methods.<br />
* Does not have to understand topology of space requires creator to specify scoring for the machine .<br />
<br />
'''Convolutional Neural Networks:'''<br />
<br />
Utilize iterative scoring between training and testing, along with '''gradient descent''' on a number of layered, weighted vectors to extract features from a complex data set. <br />
<br />
'''Vision Systems''' <br />
<br />
Take high throughput data and simplify before work is done ''ex'': 1080p @ 60fps <math display="inline">\rightarrow</math> 240x360 @ 15fps, broken into component channels Gaussian blur filter kernel applied to high-res images Edge detection via double threshold ML or CNN used past this to determine actual features<br />
<br />
==== Large Language Models ====<br />
Large language models (LLMs) are stochastic systems which attempt to capture the 'shape' of a language (a recursively enumerable language by [[wikipedia:Chomsky_hierarchy|Chomsky Hierarchy]]) by pulling successive tokens from a '''bag of words model'''; in this system, a corpus of text is transformed into a mathematical space of the tokens, which are substrings of the input language, and weights, which are a cosine similarity between tokens in a Hilbert space. In other words, a large language model is a function which descends the gradient of this space, and uses probability to arrive at what word goes next in a given completion. This mathematical loss minimization is performed sequentially as the text is generated, in what is known as '''self-attention''', which relates different positions of text sequence in order to compute a representation of the sequence. This function takes the form:<br />
<br />
<math display="block">\sum_{t=1}^T P(x_t | \vec{x}_{<t} \vec{x}_{i:j} )</math><br />
<br />
where tokens are generated until limit '''T''' is reached, and where there is a probability of another token generated as a function of previous tokens and the input. This functional setup is called an '''encoder-decoder'''. <br />
<br />
So, based on the corpus, and given an input, an encoder-decoder will try to create a response given an input. Usually, encoder-decoders are further trained by a process called '''Reinforcement Learning from Human Feedback''' (RLHF), in which human feedback is given in an iterative process until the underlying language weights prefer the trained preferences.<br />
<br />
====== To consider in regard to LLMs: ======<br />
<br />
* They are stochastic word generator machines, and contain structure, rather than problem solving logic.<br />
* They are typically computationally expensive to run.<br />
* They are typically monstrously computationally expensive to train.<br />
* If equipped with enough memory for self-attention, they become a Turing Machine with Type-0 Grammar. <br />
* They may occasionally generate incorrect information.<br />
* They '''hallucinate''', generating nonsense (verbal noise) when confronted with unexpected text not encountered in training. <br />
* Because the attention of a model is limited, LLMs perform badly with large multi-step processes which require dense context.</div><br />
<br />
== Software Toolkit ==<br />
<div class="outline"><br />
=== Python ===<br />
General purpose programming language with many libraries Interpreted language: each line is run one at a time by a virtual machine.<br />
<br />
'''Dependency Structure''' <br />
<br />
''system'' vs ''user'' python <br />
<br />
virtualenv: <code>$ python3 -m venv /path/to/new/environment</code> <br />
<br />
pip libraries <code>$ pip list outdated format=freeze | grep -v | cut -d=" " -f1 | xargs -n1 pip install -U</code><br />
<br />
==== Anaconda ====<br />
separate virtualenv system specifically for data science:<br />
<br />
* '''Spyder''' IDE with visual output <br />
* '''JupyterLabs''' notes with data visualizations <br />
* '''Orange''' visual IDE for stats exploration<br />
* <code>pandas, NumPy, and SciPy</code> libraries for data serialization and numerical work in python <br />
* <code>matplotlib</code> for data visualization <br />
* <code>scikit-learn</code> main library for ML <br />
* <code>DASK</code> distributed abstraction layer with <code>pandas</code> grammar to easily distribute python tasks into 1-1000 compute nodes <br />
* <code>PyTorch, and TensorFlow</code> deep learning and CNN generation systems massive compute overhead to train models require data map reduction and or imputation to run well</div><div class="outline"><br />
=== R language ===<br />
Statistical programming language: interpreter invokes compiled C or FORTRAN.<br />
<br />
Also works within Jupyter notebook for instant visualization, if wanted.<br />
<br />
Open-source and extended by the Comprehensive R Archive Network (CRAN), which includes extensive documentation.<br />
<br />
* <code>rmarkdown</code> format a document from R with optional LaTeXbindings <br />
* <code>tidyverse</code> <br />
** <code>dplyr</code> grammar for mass data manipulation <br />
** <code>ggplot2</code> a library for creating graphs and visualizations <br />
* <code>doparallel</code> cost-free abstraction, pooling of CPU threads <br />
* <code>mlr</code> interface to a large number of classification and regression techniques <br />
* <code>shiny</code> provides ability to create web servers similar to NodeJS or Python Flaskl<br />
<br />
=== Intel MKL (Math Kernel Library) ===<br />
Improves performance for Fast Fourier Transforms, linear algebra operations, vector math, deep neural networks, and kernel solvers. <br />
<br />
Default math backend for NumPy, SciPy, and MATLAB <br />
<br />
Not hardware agnostic: chooses slowest solvers for non-Intel chips by default <br />
<br />
=== OpenBLAS and LAPACK ===<br />
'''LAPACK''' (Linear Algebra PACKage) provides APIs much like MKL <br />
<br />
'''OpenBLAS''' (Basic Linear Algebra Subprograms) extends LAPACK with optimizations for parallel computing <br />
<br />
Default for R and Biopython<br />
<br />
=== Message Passing Interface (MPI) ===<br />
Supported by all major compilers (Intel and OpenMP implementations) <br />
<br />
An API supporting shared-memory multiprocessing provides backend for many parallel computing systems, allowing for multi-threaded access <br />
<br />
''ex'': <code>mpirun -np $NUM_PROC /path/to/coolProgram &lt; $INPUT &gt; /path/to/output</code> <br />
<br />
=== GNU Parallel ===<br />
simple vectorization of loops over processors for non-multithreaded processes <br />
<br />
''ex'': <code>parallel -j $NUM_PROC /path/to/thescript.sh ::: 1..n ::: 1..m</code><br />
<br />
=== CUDA/OpenCL ===<br />
Nvidia-specific '''CUDA''' and open-source '''OpenCL''' provide a hardware abstracting API for using GPU for compute tasks <br />
<br />
must-have for Pytorch or TensorFlow workloads <br />
<br />
Nomenclature Divergence <br />
<br />
* CUDA thread = OpenCL work item = CPU lane <br />
* CUDA multiprocessor = OpenCL compute unit = CPU<br />
<br />
=== High Performance Computers ===<br />
HPC or super-computing clusters provide high throughput analysis.<br />
<br />
Amazingly high amount of computational power.<br />
<br />
Need to plan your analysis.<br />
<br />
'''NDSU Center for Computationally-Assisted Science and Technology (CCAST)''' provides a platform for these workloads connect via <code>ssh</code> uses loadable modules: <br />
<br />
* ''ex:'' <code>module load parallel</code> <br />
<br />
batch processing via PBS scripting<br />
</div><br />
<br />
== <span>Continued Learning</span> ==<br />
<div class="outline"><br />
=== General Programming/Computers Websites ===<br />
<br />
* StackOverflow.com - check before asking new questions <br />
* RosettaCode.org - data structures and algorithms in many languages <br />
* Linux.die.net/man/ - the Linux manual <br />
* grymoire.com/Unix/ - more *nix CLI tutorials<br />
<br />
=== Python ===<br />
<br />
* docs.python.org/3/ - the official python documentation <br />
* docs.python.org/3/tutorial - the official tutorial <br />
* diveintopython.net - guided tutorial online <br />
* pythontutor.com - visual debugger <br />
<br />
=== R ===<br />
<br />
* cran.r-project.org - CRAN <br />
* cran.r-project.org/manuals.html <br />
* rdrr.io - meta-manual lookup and many other tools for R <br />
* swirlstats.com - learn R, in R <br />
* statlearning.com - statistical machine learning coursework<br />
<br />
=== Recommended Reading ===<br />
<br />
* ''An Introduction to Statistical Machine Learning'' by Gareth James et al. <br />
* ''A Primer on Scientific Programming with Python'' by Hans Petter Langtangen <br />
* ''R for Data Science'' by Wickham and Grolmund</div></div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Ffmpeg&diff=150Ffmpeg2022-11-22T23:31:31Z<p>Sysadmin: not computer code</p>
<hr />
<div>ffmpeg is a free and open-source suite of programs which allow for the conversion and creation of video, audio, and image files. While ffmpeg can be thought of as a swiss-army knife for multimedia formats, we can use ffmpeg on the output of VMD (or any other MD sim) to create smooth animations. <br />
==Installation==<br />
Linux and Mac OSX users can use the built-in package manager to install ffmpeg via command line:<br />
<br />
* Ubuntu/Debian Linux: <code># sudo apt install ffmpeg</code><br />
* Mac OSX via homebrew: <code>$ brew install ffmpeg --with-fdk-aac --with-ffplay --with-freetype --with-libass --with-libquvi --with-libvorbis --with-libvpx --with-opus --with-x265</code> <br />
<br />
Windows users can install manually using packages built by community package maintainers: <br />
<br />
* Download 'release' package build archive from https://www.gyan.dev/ffmpeg/builds/ <br />
* Unzip into an easy-to-find folder, such as <code>C:\ffmpeg</code><br />
* To add ffmpeg to your system <code>PATH</code>, so that you can run it from any window, open a CMD/Powershell window as Administrator, then run: <code>setx /M PATH "%PATH%;C:\ffmpeg\bin"</code><br />
<br />
In any case, ffmpeg can now be invoked from a command-line. To make sure we have a working system, you can run: <code>ffmpeg -version</code> to display information about the build. <br />
<br />
== Use with VMD ==<br />
ffmpeg is already linked against VMD in the Unix versions of the program. For Windows users, Navigating to 'Extensions > Movie Maker' will allow the creation of a number of images, one per step size specified, which are saved to a specified folder. After creating these images, you can perform the following to create a movie file: <br />
<br />
* change to the directory: <code>cd C:\output_folder\</code><br />
* to make an MP4 with 30 frames a second from images in the format starting 'output.0001.bmp': <code>ffmpeg -r 30 -crf 25 -i output.%04d.bmp yourmovietitle.mp4</code><br />
* to make a WMV with the same settings: <code>ffmpeg -r 30 -crf 25 -i output.%04d.bmp yourmovietitle.mpv</code><br />
<br />
ffmpeg will read sequentially across the pattern %04d as input -i, and create a movie with rate -r and quality -crf. Many other options are possible, and would require a look at the ffmpeg documentation. <br />
<br />
== Outside Resource ==<br />
ffmpeg homepage wiki: https://trac.ffmpeg.org/wiki</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Ffmpeg&diff=149Ffmpeg2022-11-22T23:29:49Z<p>Sysadmin: quote mark typo edit</p>
<hr />
<div>ffmpeg is a free and open-source suite of programs which allow for the conversion and creation of video, audio, and image files. While ffmpeg can be thought of as a swiss-army knife for multimedia formats, we can use ffmpeg on the output of VMD (or any other MD sim) to create smooth animations. <br />
==Installation==<br />
Linux and Mac OSX users can use the built-in package manager to install ffmpeg via command line:<br />
<br />
* Ubuntu/Debian Linux: <code># sudo apt install ffmpeg</code><br />
* Mac OSX via homebrew: <code>$ brew install ffmpeg --with-fdk-aac --with-ffplay --with-freetype --with-libass --with-libquvi --with-libvorbis --with-libvpx --with-opus --with-x265</code> <br />
<br />
Windows users can install manually using packages built by community package maintainers: <br />
<br />
* Download 'release' package build archive from https://www.gyan.dev/ffmpeg/builds/ <br />
* Unzip into an easy-to-find folder, such as <code>C:\ffmpeg</code><br />
* To add ffmpeg to your system <code>PATH</code>, so that you can run it from any window, open a CMD/Powershell window as Administrator, then run: <code>setx /M PATH "%PATH%;C:\ffmpeg\bin"</code><br />
<br />
In any case, ffmpeg can now be invoked from a command-line. To make sure we have a working system, you can run: <code>ffmpeg -version</code> to display information about the build. <br />
<br />
== Use with VMD ==<br />
ffmpeg is already linked against VMD in the Unix versions of the program. For Windows users, Navigating to 'Extensions > Movie Maker' will allow the creation of a number of images, one per step size specified, which are saved to a specified folder. After creating these images, you can perform the following to create a movie file: <br />
<br />
* change to the directory: <code>cd C:\output_folder\</code><br />
* to make an MP4 with 30 frames a second from images in the format starting <code>output.0001.bmp</code>: <code>ffmpeg -r 30 -crf 25 -i output.%04d.bmp yourmovietitle.mp4</code><br />
* to make a WMV with the same settings: <code>ffmpeg -r 30 -crf 25 -i output.%04d.bmp yourmovietitle.mpv</code><br />
<br />
ffmpeg will read sequentially across the pattern %04d as input -i, and create a movie with rate -r and quality -crf. Many other options are possible, and would require a look at the ffmpeg documentation. <br />
<br />
== Outside Resource ==<br />
ffmpeg homepage wiki: https://trac.ffmpeg.org/wiki</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Main_Page&diff=148Main Page2022-11-08T20:49:17Z<p>Sysadmin: </p>
<hr />
<div>=Welcome to the Rasulev Research Group Wiki!=<br />
==The Computational Polymer Science and Cheminformatics Group at North Dakota State University==<br />
This page hosts the tutorials and procedures for the group. Implementing MediaWiki, it provides a platform to organize knowledge and make it available to both lab collaborators and students. <br />
<br />
=== [[File:OOjs UI icon edit-ltr.svg|20px|middle|link=]]Topic Crash Courses ===<br />
* [[Statistical Machine Learning]]<br />
* [[Unix crash course|Unix Crash Course]]<br />
===[[File:OOjs UI icon chem.svg|20px|middle|link=]]Lab Procedures and Software Tutorials===<br />
<div class="col"><br />
* [[ADF ReaxFF]]<br />
* [[Avogadro Procedure|Avogadro]]<br />
* [[BuildQsar Procedure|BuildQSAR]]<br />
* [[ChemDraw Procedure|ChemDraw]]<br />
* [[ChemSketchProcedure|ChemSketch]]<br />
* [[Ffmpeg|Creating animations with ffmpeg]]<br />
* [[Dragon5 Procedure|Dragon5]]<br />
* [[Generating Descriptors Workflow]]<br />
* [[HyperChem]]<br />
* [[Jupyterlab anaconda|JupyterLab]]<br />
* [[OpenBabelGUI]]<br />
* [[QSARINS Procedure|QSARINS]]<br />
* [[VASP and VMD on CCAST|VASP]]<br />
* [[WEKA Steps for Loading Data|WEKA]]<br />
</div><br />
===[[File:OOjs UI icon advanced.svg|20px|middle|link=]]Benchmarks and Findings===<br />
* [[Benchmarking NAMD workflows for GPU containers]]<br />
<br />
__NOEDITSECTION__<br />
__NOTOC__</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Template:Div_col&diff=147Template:Div col2022-11-08T20:33:01Z<p>Sysadmin: created div column css</p>
<hr />
<div>/* {{pp|small=yes}} */<br />
.div-col {<br />
margin-top: 0.3em;<br />
column-width: 30em;<br />
}<br />
<br />
.div-col-rules {<br />
column-rule: 1px solid #aaa;<br />
}<br />
<br />
/* Reset top margin for lists in div col */<br />
.div-col dl,<br />
.div-col ol,<br />
.div-col ul {<br />
margin-top: 0;<br />
}<br />
<br />
/* Avoid elements breaking between columns<br />
See also Template:No col break */<br />
.div-col li,<br />
.div-col dd {<br />
page-break-inside: avoid; /* Removed from CSS in favor of break-inside c. 2020 */<br />
break-inside: avoid-column;<br />
}</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Main_Page&diff=146Main Page2022-11-08T19:42:10Z<p>Sysadmin: </p>
<hr />
<div><div><br />
=Welcome to the Rasulev Research Group Wiki!=<br />
==The Computational Polymer Science and Cheminformatics Group at North Dakota State University==<br />
<br />
This page hosts the tutorials and procedures for the group. Implementing MediaWiki, it provides a platform to organize knowledge and make it available to both lab collaborators and students. <br />
<div style="clear: both;"></div><br />
<br />
=== [[File:OOjs UI icon edit-ltr.svg|20px|middle|link=]]Topic Crash Courses ===<br />
<br />
* [[Statistical Machine Learning]]<br />
* [[Unix crash course|Unix Crash Course]]<div id="audiences" class="mainpage_row"><div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon chem.svg|20px|middle|link=]]</span><span>Lab Procedures and Software Tutorials</span></h3><br />
<div><br />
* [[ADF ReaxFF]]<br />
* [[Avogadro Procedure|Avogadro]]<br />
* [[BuildQsar Procedure|BuildQSAR]]<br />
* [[ChemDraw Procedure|ChemDraw]]<br />
* [[ChemSketchProcedure|ChemSketch]]<br />
* [[Ffmpeg|Creating animations with ffmpeg]]<br />
* [[Dragon5 Procedure|Dragon5]]<br />
* [[Generating Descriptors Workflow]]<br />
* [[HyperChem]]<br />
* [[Jupyterlab anaconda|JupyterLab]]<br />
* [[OpenBabelGUI]]<br />
* [[QSARINS Procedure|QSARINS]]<br />
* [[VASP and VMD on CCAST|VASP]]<br />
* [[WEKA Steps for Loading Data|WEKA]]<br />
</div><br />
</div><br />
</div><div id="audiences" class="mainpage_row"><br />
<div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon advanced.svg|20px|middle|link=]]</span><span>Benchmarks and Findings</span></h3><br />
<div><br />
* [[Benchmarking NAMD workflows for GPU containers]]<br />
</div><br />
</div><br />
</div><br />
__NOEDITSECTION__<br />
__NOTOC__</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Ffmpeg&diff=145Ffmpeg2022-11-08T19:41:01Z<p>Sysadmin: Created FFMPEG tutorial page</p>
<hr />
<div>ffmpeg is a free and open-source suite of programs which allow for the conversion and creation of video, audio, and image files. While ffmpeg can be thought of as a swiss-army knife for multimedia formats, we can use ffmpeg on the output of VMD (or any other MD sim) to create smooth animations. <br />
==Installation==<br />
Linux and Mac OSX users can use the built-in package manager to install ffmpeg via command line:<br />
<br />
* Ubuntu/Debian Linux: <code># sudo apt install ffmpeg</code><br />
* Mac OSX via homebrew: <code>$ brew install ffmpeg --with-fdk-aac --with-ffplay --with-freetype --with-libass --with-libquvi --with-libvorbis --with-libvpx --with-opus --with-x265</code> <br />
<br />
Windows users can install manually using packages built by community package maintainers: <br />
<br />
* Download 'release' package build archive from https://www.gyan.dev/ffmpeg/builds/ <br />
* Unzip into an easy-to-find folder, such as <code>C:\ffmpeg</code><br />
* To add ffmpeg to your system <code>PATH</code>, so that you can run it from any window, open a CMD/Powershell window as Administrator, then run: <code>setx /M PATH "%PATH%;C:\ffmpeg\bin</code><br />
<br />
In any case, ffmpeg can now be invoked from a command-line. To make sure we have a working system, you can run: <code>ffmpeg -version</code> to display information about the build. <br />
<br />
== Use with VMD ==<br />
ffmpeg is already linked against VMD in the Unix versions of the program. For Windows users, Navigating to 'Extensions > Movie Maker' will allow the creation of a number of images, one per step size specified, which are saved to a specified folder. After creating these images, you can perform the following to create a movie file: <br />
<br />
* change to the directory: <code>cd C:\output_folder\</code><br />
* to make an MP4 with 30 frames a second from images in the format starting <code>output.0001.bmp</code>: <code>ffmpeg -r 30 -crf 25 -i output.%04d.bmp yourmovietitle.mp4</code><br />
* to make a WMV with the same settings: <code>ffmpeg -r 30 -crf 25 -i output.%04d.bmp yourmovietitle.mpv</code><br />
<br />
ffmpeg will read sequentially across the pattern %04d as input -i, and create a movie with rate -r and quality -crf. Many other options are possible, and would require a look at the ffmpeg documentation. <br />
<br />
== Outside Resource ==<br />
ffmpeg homepage wiki: https://trac.ffmpeg.org/wiki</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Statistical_Machine_Learning&diff=144Statistical Machine Learning2022-11-08T19:16:12Z<p>Sysadmin: /* GNU Parallel */</p>
<hr />
<div>=== <span>Overview</span> ===<br />
<div class="outline"><br />
A crash course to enable gentle introduction into the machine learning techniques and its applications into data science. <br />
<br />
Topics Covered: <br />
<br />
* Basic Concepts <br />
* Data Models <br />
* Machine Learning <br />
* ML Techniques <br />
* Software Toolkit <br />
* Continued Learning<br />
<br />
== <span>Basic Concepts</span> ==<br />
<div class="outline"><br />
=== Definitions: ===<br />
<br />
* '''Statistical Machine Learning''' is a set of tools used to model and understand complex data sets <br />
* '''Data Science''' is a set of techniques in computing to support the analysis of data <br />
** Not very useful without some domain knowledge: it is important to ''know your data''. <br />
* Includes analytic techniques: <br />
** descriptive statistics <br />
** data visualization <br />
** statistical machine learning <br />
** neural networks<br />
** actor-environment models <br />
* Also includes computational techniques: <br />
** database administration <br />
** management of information systems <br />
** parallelization <math display="inline">\rightarrow</math> high performance computing<br />
</div><br />
<br />
=== Basic Concepts ===<br />
<div class="outline"><br />
==== Knowing your data ====<br />
Technical definition: <br />
<br />
Let <math display="inline">n</math> represent a number of distinct '''observations''', and let <math display="inline">p</math> represent a number of '''predictors''' Then, our observed data <math display="inline">\textbf{X}</math> is an <math display="inline">n\times p</math> matrix with row observation vectors <math display="inline">\vec{x}_{1..n}</math> and column predictor vectors <math display="inline">\vec{x}_{1..p}</math>. <br />
<br />
In addition, we will also have '''response''' variable(s) <math display="inline">\textbf{Y}</math>, which is a made up of some <math display="inline">n</math>-length vectors. <br />
<br />
So, our combined dataset consists of <math>[(\vec{x}_1,\vec{y}_1),(\vec{x}_2, \vec{y}_2),...,(\vec{x}_n,\vec{y}_n)]</math><br />
<br />
'''Our Mission''': determine relationships between <math display="inline">\textbf{X}</math> and <math display="inline">\textbf{Y}</math> which are mathematically sound, leading to better understanding<br />
<br />
Typically a table has columns as features, rows as entries<br />
<br />
'''Entries''' might be '''numeric''' or '''categorical'''. <br />
<br />
Data sources are either '''Structured''' or '''Unstructured''': <br />
<br />
* Unstructured data will require some transformation. <br />
<br />
Some data may also be '''time series''' taking a sampling of points over time, contributing to a 3-dimensional '''Data Cub'''<br />
<br />
Several techniques can be used to reduce complex data: <br />
<br />
* '''numeric representation''' mapping of categorical information into numbers. <br />
* '''scaling''' redefine a new range for a predictor vector. <br />
* '''normalization''' redefine a predictor by its mean and standard deviation, giving a normal distribution of values. <br />
* '''dimension reduction''' lose fine grain of data, but gain understandability. <br />
* '''feature extraction''' a data mining technique in which we can generate new predictors from known information<br />
<br />
==== Modeling ====<br />
What is a model?<br />
<br />
'''Very well-known model''': Gravity is a functional model between masses, distances, and force. <math display="block">F = G\frac{m_1m_2}{r^2} \rightarrow g = \frac{G M}{r^2} \rightarrow v(t) = v(0) - gt.</math> '''<math display="block">k = Ae^{\frac{E_a}{K_b T}}.</math>Statistics definition''': <br />
<br />
Let <math display="inline">X = (\vec{x}_1, \vec{x}_2, ..., \vec{x}_p)</math> each of length <math display="inline">n</math>, and <math display="inline">Y = (\vec{y})</math> of length <math display="inline">n</math>, then for <math display="inline">X</math> and <math display="inline">Y</math>, there exists a function with a '''systematic''' <math display="inline">f</math> and '''error term''' <math display="inline">\varepsilon</math>: <math display="block">Y = f(X) + \varepsilon</math>Why do we even estimate <math display="inline">f</math> at all? '''Prediction''' or '''Inference'''<br />
<br />
'''Predictive models''' create an estimator <math display="inline">\hat{f}</math> which we can use to estimate <math display="inline">Y</math> using a sample <math display="inline">X</math> from a larger population: <math display="block">\hat{ Y} = \hat{f}(X)</math> With error: <math display="inline">E(Y - \hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \varepsilon</math>. <br />
<br />
'''Inference models''' are primarily interested in how <math display="inline">Y</math> is affected by <math display="inline">X</math>: <br />
<br />
* What predictors associated with response? <br />
* What is the relationship of predictors to response? <br />
* What is the overall nature of relationship between <math display="inline">Y</math> and the predictors.<br />
<br />
==== Signal vs Noise ====<br />
Consider '''precision''' and '''accuracy'''. <br />
<br />
* Both contribute into data set <br />
<br />
'''noise:''' variation in data which detracts from constructing '''information''', as opposed to '''signal'''–data which is representative of a system under study and contains information. <br />
<br />
High signal to noise allows us to minimize '''reducible error''', caused by sampling technique. <br />
<br />
Different than '''irreducible error''', created by factors we are not measuring.<br />
<br />
==== Error and Fit ====<br />
In Modeling In the terms of modeling, precision of a model is referred to '''variance''' and the accuracy of a model its degree of '''bias'''. <br />
<br />
Generally, overly complex models generate high variance, and can '''over-fit''' to input data, making the model useless to new data.<br />
<br />
Generally, '''Mean Square Error''' or MSE, used to determine goodness-of-fit for model calibration: <math display="inline">MSE = \frac{1}{n}\sum^n_{i=1}(y_i - \hat{f}(x_i))^2</math>. <br />
<br />
Error rate used in classification: <math display="inline">\frac{1}{n} \sum^n_{i=1} I(y_i \ne \hat{y}_i)</math>. <br />
<br />
However for reporting, <math display="inline">R^2</math> statistic is more often used, because it gives a value between 0 and 1 useful to determine how much of variance in <math display="inline">Y</math> is explained by variance in <math display="inline">X</math>: <math display="inline">R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2} = \frac{RSS}{TSS}</math><br />
<br />
Example functions relating <math display="inline">X</math> and <math display="inline">Y</math>:<br />
<br />
* A linear function: <math display="inline">Y = \beta_0 + \beta_1 X + \varepsilon</math> <br />
* A polynomial: <math display="inline">Y = \beta_0 + \beta_1 X + \beta_2 X + ... + \varepsilon</math> <br />
* A natural function: <math display="inline">Y = e^{-\alpha_0 X^{\alpha_1}} + \varepsilon</math> <br />
* A logistical function: <math display="inline">p = \frac{ e^{\beta_0 + \beta_1 X} }{1 + e^{\beta_0 + \beta_1 X} }</math> <br />
* A series of nested <code>if</code> statements <br />
* A series of differential equations: <math display="inline">x' = x_n - \bar{x}_n: y' = y_n - \beta_n x_n'</math><br />
<br />
==== Why Machine Learning? ====<br />
Types of Questions: <br />
<br />
* '''Exact Solution is known''' normal coding problems, linear models, and classical statistics. <br />
* '''Exact Solution is unknown, but can be extracted with work''' work with systems experts and domain knowledge to create code. <br />
* '''Exact Solution is known, but not yet conveyable''' ML is useful. <br />
* '''Exact Solution not known by humans''' ML and/or Deep Learning needed<br />
<br />
''Example:'' consider a prediction of temperature: <br />
<br />
* '''Knowledge Based Models''' <br />
** Physics and atmospheric science based model. <br />
** Up to differential equations on chaotic systems. <br />
** As fine granularity of prediction increases, number of factors and density of data quickly becomes too much for most humans to consider <br />
* '''Data Driven Models''' <br />
** Use ML to provide '''iterative''' gain to reduce error <br />
*** Known data→model creation→point toward new factors.<br />
** Uses a split in training and testing data, or sum of error to move toward the correct answer.<br />
** Human researchers more free to find more data, improve prediction, develop theories<br />
<br />
== <span>Techniques</span> ==<br />
<br />
=== Machine Learning in General ===<br />
'''Supervised Learning:''' the model estimates, error verifies <br />
<br />
* If incorrect, needs user input for correction.<br />
* ''Example'': a computer vision system trained to find features in images via user annotated images. <br />
<br />
'''Unsupervised Learning:''' Clustering/Grouping of similar items<br />
<br />
* Need a similarity measure via feature vectors and ability to adjust weights <br />
* ''Example'': Taste prediction algorithms used in web advertising. <br />
<br />
'''Reinforced Learning''' Model estimates a sequence of guesses <br />
<br />
* Correct if and only if the entire sequence or a parameterized output scoring<br />
* Instant feedback but high compute cost <br />
* ''Example'': Game-play in actor-environment model<br />
<br />
=== Types of Problems and Output ===<br />
'''Numeric''' function maps <math display="inline">X</math> to <math display="inline">Y</math> and output is in <math display="inline">\mathbb{R}</math>. <br />
<br />
'''Categorization''' non-orderable sorting <br />
<br />
'''Clustering''' finding principle ways groups differ <br />
<br />
'''Anomaly Detection''' finding data points which are out of the ordinary <br />
<br />
'''Actor Models''' real-time decision making or detailed simulation<br />
<br />
=== Predictive Models ===<br />
Predictive ML models which are also Linear: <br />
<br />
* Utilize a split of training and test data: test-training or <math display="inline">k</math>-fold cross-validation <br />
* use a function mapping of one or more independent variables to the dependent variables, then re-evaluate to reduce error Includes techniques for mixed model reduction<br />
* Reduction of the number of predictors via '''Lasso''', '''Ridge''', and '''Elastic Net''' techniques<br />
<br />
=== Feature-vector based models ===<br />
Nested <code>if</code> statements try to find decision boundaries by distance between independent data and dependent outcome features have weighted probability most information by Bayesian inference<br />
<br />
* '''Decision Trees''' can be used to create decision models for linearly separable data effectively a neural network with one neuron<br />
* '''Random Forest''' utilizes a number of differently-tuned trees trees provide consensus voting-based approach for non-linearly separable data smaller tree depth typically prevents over-fit<br />
<br />
=== Clustering ===<br />
Groups data into cluster such that distance within clusters is small, and between differing groups is large <br />
<br />
Works with any well-defined &quot;distance&quot; function: Euclidean, Hamming, Inner Product, etc. <br />
<br />
'''k-Means Clustering:''' <br />
<br />
* choose <math display="inline">k</math> number of clusters randomly distribute <math display="inline">k</math> points, '''centroids''', into feature space <br />
* divide and classify data by distance to <math display="inline">k</math> centroids <br />
* move centroids based on center of groups repeats until convergence to some epsilon value<br />
* where points no longer move across iterations <br />
* '''Goodness of Fit''' for <math display="inline">N-M</math> possible values of <math display="inline">k</math>, an inflection in overall likelihood ratio given by probability function for set<br />
<br />
'''K-nearest Neighbors:''' <br />
<br />
Creates a probabilistic decision boundary within a feature space between <math display="inline">K</math> centroids<br />
<br />
Unsupervised system to find structures of data works on majority voting system<br />
<br />
'''Gradient Boosting:'''<br />
<br />
System attempts to find the direction and vector of change in a dimensional field, and follow these iteratively to find local extrema. <br />
<br />
Most use some '''Quasi-Newton Method''' for finding extrema for faster centroid convergence. <br />
<br />
'''Support Vector Machine:''' <br />
<br />
Also known as SVM, applies a classifier into high-dimensional data to split points into groups some use a '''kernel trick.'''<br />
<br />
Generates the inner product space of two arbitrary-dimensional numeric matrix spaces, showing the shape of data<br />
<br />
=== Genetic Algorithms ===<br />
Utilize some adversarial scoring method of initially randomized vectors:<br />
<br />
* ’survivors’ become the basis of new models similar iterative concept to gradient methods.<br />
* Does not have to understand topology of space requires creator to specify scoring for the machine .<br />
<br />
'''Convolutional Neural Networks:'''<br />
<br />
Utilize iterative scoring between training and testing, along with '''gradient descent''' on a number of layered, weighted vectors to extract features from a complex data set. <br />
<br />
'''Vision Systems''' <br />
<br />
Take high throughput data and simplify before work is done ''ex'': 1080p @ 60fps <math display="inline">\rightarrow</math> 240x360 @ 15fps, broken into component channels Gaussian blur filter kernel applied to high-res images Edge detection via double threshold ML or CNN used past this to determine actual features<br />
</div><br />
<br />
== Software Toolkit ==<br />
<div class="outline"><br />
=== Python ===<br />
General purpose programming language with many libraries Interpreted language: each line is run one at a time by a virtual machine.<br />
<br />
'''Dependency Structure''' <br />
<br />
''system'' vs ''user'' python <br />
<br />
virtualenv: <code>$ python3 -m venv /path/to/new/environment</code> <br />
<br />
pip libraries <code>$ pip list outdated format=freeze | grep -v | cut -d=" " -f1 | xargs -n1 pip install -U</code><br />
<br />
==== Anaconda ====<br />
separate virtualenv system specifically for data science:<br />
<br />
* '''Spyder''' IDE with visual output <br />
* '''JupyterLabs''' notes with data visualizations <br />
* '''Orange''' visual IDE for stats exploration<br />
* <code>pandas, NumPy, and SciPy</code> libraries for data serialization and numerical work in python <br />
* <code>matplotlib</code> for data visualization <br />
* <code>scikit-learn</code> main library for ML <br />
* <code>DASK</code> distributed abstraction layer with <code>pandas</code> grammar to easily distribute python tasks into 1-1000 compute nodes <br />
* <code>PyTorch, and TensorFlow</code> deep learning and CNN generation systems massive compute overhead to train models require data map reduction and or imputation to run well</div><div class="outline"><br />
=== R language ===<br />
Statistical programming language: interpreter invokes compiled C or FORTRAN.<br />
<br />
Also works within Jupyter notebook for instant visualization, if wanted.<br />
<br />
Open-source and extended by the Comprehensive R Archive Network (CRAN), which includes extensive documentation.<br />
<br />
* <code>rmarkdown</code> format a document from R with optional LaTeXbindings <br />
* <code>tidyverse</code> <br />
** <code>dplyr</code> grammar for mass data manipulation <br />
** <code>ggplot2</code> a library for creating graphs and visualizations <br />
* <code>doparallel</code> cost-free abstraction, pooling of CPU threads <br />
* <code>mlr</code> interface to a large number of classification and regression techniques <br />
* <code>shiny</code> provides ability to create web servers similar to NodeJS or Python Flaskl<br />
<br />
=== Intel MKL (Math Kernel Library) ===<br />
Improves performance for Fast Fourier Transforms, linear algebra operations, vector math, deep neural networks, and kernel solvers. <br />
<br />
Default math backend for NumPy, SciPy, and MATLAB <br />
<br />
Not hardware agnostic: chooses slowest solvers for non-Intel chips by default <br />
<br />
=== OpenBLAS and LAPACK ===<br />
'''LAPACK''' (Linear Algebra PACKage) provides APIs much like MKL <br />
<br />
'''OpenBLAS''' (Basic Linear Algebra Subprograms) extends LAPACK with optimizations for parallel computing <br />
<br />
Default for R and Biopython<br />
<br />
=== Message Passing Interface (MPI) ===<br />
Supported by all major compilers (Intel and OpenMP implementations) <br />
<br />
An API supporting shared-memory multiprocessing provides backend for many parallel computing systems, allowing for multi-threaded access <br />
<br />
''ex'': <code>mpirun -np $NUM_PROC /path/to/coolProgram &lt; $INPUT &gt; /path/to/output</code> <br />
<br />
=== GNU Parallel ===<br />
simple vectorization of loops over processors for non-multithreaded processes <br />
<br />
''ex'': <code>parallel -j $NUM_PROC /path/to/thescript.sh ::: 1..n ::: 1..m</code><br />
<br />
=== CUDA/OpenCL ===<br />
Nvidia-specific '''CUDA''' and open-source '''OpenCL''' provide a hardware abstracting API for using GPU for compute tasks <br />
<br />
must-have for Pytorch or TensorFlow workloads <br />
<br />
Nomenclature Divergence <br />
<br />
* CUDA thread = OpenCL work item = CPU lane <br />
* CUDA multiprocessor = OpenCL compute unit = CPU<br />
<br />
=== High Performance Computers ===<br />
HPC or super-computing clusters provide high throughput analysis.<br />
<br />
Amazingly high amount of computational power.<br />
<br />
Need to plan your analysis.<br />
<br />
'''NDSU Center for Computationally-Assisted Science and Technology (CCAST)''' provides a platform for these workloads connect via <code>ssh</code> uses loadable modules: <br />
<br />
* ''ex:'' <code>module load parallel</code> <br />
<br />
batch processing via PBS scripting<br />
</div><br />
<br />
== <span>Continued Learning</span> ==<br />
<div class="outline"><br />
=== General Programming/Computers Websites ===<br />
<br />
* StackOverflow.com - check before asking new questions <br />
* RosettaCode.org - data structures and algorithms in many languages <br />
* Linux.die.net/man/ - the Linux manual <br />
* grymoire.com/Unix/ - more *nix CLI tutorials<br />
<br />
=== Python ===<br />
<br />
* docs.python.org/3/ - the official python documentation <br />
* docs.python.org/3/tutorial - the official tutorial <br />
* diveintopython.net - guided tutorial online <br />
* pythontutor.com - visual debugger <br />
<br />
=== R ===<br />
<br />
* cran.r-project.org - CRAN <br />
* cran.r-project.org/manuals.html <br />
* rdrr.io - meta-manual lookup and many other tools for R <br />
* swirlstats.com - learn R, in R <br />
* statslearning.com - statistical machine learning coursework<br />
<br />
=== Recommended Reading ===<br />
<br />
* ''An Introduction to Statistical Machine Learning'' by Gareth James et al. <br />
* ''A Primer on Scientific Programming with Python'' by Hans Petter Langtangen <br />
* ''R for Data Science'' by Wickham and Grolmund</div></div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Statistical_Machine_Learning&diff=143Statistical Machine Learning2022-11-03T21:08:20Z<p>Sysadmin: </p>
<hr />
<div>=== <span>Overview</span> ===<br />
<div class="outline"><br />
A crash course to enable gentle introduction into the machine learning techniques and its applications into data science. <br />
<br />
Topics Covered: <br />
<br />
* Basic Concepts <br />
* Data Models <br />
* Machine Learning <br />
* ML Techniques <br />
* Software Toolkit <br />
* Continued Learning<br />
<br />
== <span>Basic Concepts</span> ==<br />
<div class="outline"><br />
=== Definitions: ===<br />
<br />
* '''Statistical Machine Learning''' is a set of tools used to model and understand complex data sets <br />
* '''Data Science''' is a set of techniques in computing to support the analysis of data <br />
** Not very useful without some domain knowledge: it is important to ''know your data''. <br />
* Includes analytic techniques: <br />
** descriptive statistics <br />
** data visualization <br />
** statistical machine learning <br />
** neural networks<br />
** actor-environment models <br />
* Also includes computational techniques: <br />
** database administration <br />
** management of information systems <br />
** parallelization <math display="inline">\rightarrow</math> high performance computing<br />
</div><br />
<br />
=== Basic Concepts ===<br />
<div class="outline"><br />
==== Knowing your data ====<br />
Technical definition: <br />
<br />
Let <math display="inline">n</math> represent a number of distinct '''observations''', and let <math display="inline">p</math> represent a number of '''predictors''' Then, our observed data <math display="inline">\textbf{X}</math> is an <math display="inline">n\times p</math> matrix with row observation vectors <math display="inline">\vec{x}_{1..n}</math> and column predictor vectors <math display="inline">\vec{x}_{1..p}</math>. <br />
<br />
In addition, we will also have '''response''' variable(s) <math display="inline">\textbf{Y}</math>, which is a made up of some <math display="inline">n</math>-length vectors. <br />
<br />
So, our combined dataset consists of <math>[(\vec{x}_1,\vec{y}_1),(\vec{x}_2, \vec{y}_2),...,(\vec{x}_n,\vec{y}_n)]</math><br />
<br />
'''Our Mission''': determine relationships between <math display="inline">\textbf{X}</math> and <math display="inline">\textbf{Y}</math> which are mathematically sound, leading to better understanding<br />
<br />
Typically a table has columns as features, rows as entries<br />
<br />
'''Entries''' might be '''numeric''' or '''categorical'''. <br />
<br />
Data sources are either '''Structured''' or '''Unstructured''': <br />
<br />
* Unstructured data will require some transformation. <br />
<br />
Some data may also be '''time series''' taking a sampling of points over time, contributing to a 3-dimensional '''Data Cub'''<br />
<br />
Several techniques can be used to reduce complex data: <br />
<br />
* '''numeric representation''' mapping of categorical information into numbers. <br />
* '''scaling''' redefine a new range for a predictor vector. <br />
* '''normalization''' redefine a predictor by its mean and standard deviation, giving a normal distribution of values. <br />
* '''dimension reduction''' lose fine grain of data, but gain understandability. <br />
* '''feature extraction''' a data mining technique in which we can generate new predictors from known information<br />
<br />
==== Modeling ====<br />
What is a model?<br />
<br />
'''Very well-known model''': Gravity is a functional model between masses, distances, and force. <math display="block">F = G\frac{m_1m_2}{r^2} \rightarrow g = \frac{G M}{r^2} \rightarrow v(t) = v(0) - gt.</math> '''<math display="block">k = Ae^{\frac{E_a}{K_b T}}.</math>Statistics definition''': <br />
<br />
Let <math display="inline">X = (\vec{x}_1, \vec{x}_2, ..., \vec{x}_p)</math> each of length <math display="inline">n</math>, and <math display="inline">Y = (\vec{y})</math> of length <math display="inline">n</math>, then for <math display="inline">X</math> and <math display="inline">Y</math>, there exists a function with a '''systematic''' <math display="inline">f</math> and '''error term''' <math display="inline">\varepsilon</math>: <math display="block">Y = f(X) + \varepsilon</math>Why do we even estimate <math display="inline">f</math> at all? '''Prediction''' or '''Inference'''<br />
<br />
'''Predictive models''' create an estimator <math display="inline">\hat{f}</math> which we can use to estimate <math display="inline">Y</math> using a sample <math display="inline">X</math> from a larger population: <math display="block">\hat{ Y} = \hat{f}(X)</math> With error: <math display="inline">E(Y - \hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \varepsilon</math>. <br />
<br />
'''Inference models''' are primarily interested in how <math display="inline">Y</math> is affected by <math display="inline">X</math>: <br />
<br />
* What predictors associated with response? <br />
* What is the relationship of predictors to response? <br />
* What is the overall nature of relationship between <math display="inline">Y</math> and the predictors.<br />
<br />
==== Signal vs Noise ====<br />
Consider '''precision''' and '''accuracy'''. <br />
<br />
* Both contribute into data set <br />
<br />
'''noise:''' variation in data which detracts from constructing '''information''', as opposed to '''signal'''–data which is representative of a system under study and contains information. <br />
<br />
High signal to noise allows us to minimize '''reducible error''', caused by sampling technique. <br />
<br />
Different than '''irreducible error''', created by factors we are not measuring.<br />
<br />
==== Error and Fit ====<br />
In Modeling In the terms of modeling, precision of a model is referred to '''variance''' and the accuracy of a model its degree of '''bias'''. <br />
<br />
Generally, overly complex models generate high variance, and can '''over-fit''' to input data, making the model useless to new data.<br />
<br />
Generally, '''Mean Square Error''' or MSE, used to determine goodness-of-fit for model calibration: <math display="inline">MSE = \frac{1}{n}\sum^n_{i=1}(y_i - \hat{f}(x_i))^2</math>. <br />
<br />
Error rate used in classification: <math display="inline">\frac{1}{n} \sum^n_{i=1} I(y_i \ne \hat{y}_i)</math>. <br />
<br />
However for reporting, <math display="inline">R^2</math> statistic is more often used, because it gives a value between 0 and 1 useful to determine how much of variance in <math display="inline">Y</math> is explained by variance in <math display="inline">X</math>: <math display="inline">R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2} = \frac{RSS}{TSS}</math><br />
<br />
Example functions relating <math display="inline">X</math> and <math display="inline">Y</math>:<br />
<br />
* A linear function: <math display="inline">Y = \beta_0 + \beta_1 X + \varepsilon</math> <br />
* A polynomial: <math display="inline">Y = \beta_0 + \beta_1 X + \beta_2 X + ... + \varepsilon</math> <br />
* A natural function: <math display="inline">Y = e^{-\alpha_0 X^{\alpha_1}} + \varepsilon</math> <br />
* A logistical function: <math display="inline">p = \frac{ e^{\beta_0 + \beta_1 X} }{1 + e^{\beta_0 + \beta_1 X} }</math> <br />
* A series of nested <code>if</code> statements <br />
* A series of differential equations: <math display="inline">x' = x_n - \bar{x}_n: y' = y_n - \beta_n x_n'</math><br />
<br />
==== Why Machine Learning? ====<br />
Types of Questions: <br />
<br />
* '''Exact Solution is known''' normal coding problems, linear models, and classical statistics. <br />
* '''Exact Solution is unknown, but can be extracted with work''' work with systems experts and domain knowledge to create code. <br />
* '''Exact Solution is known, but not yet conveyable''' ML is useful. <br />
* '''Exact Solution not known by humans''' ML and/or Deep Learning needed<br />
<br />
''Example:'' consider a prediction of temperature: <br />
<br />
* '''Knowledge Based Models''' <br />
** Physics and atmospheric science based model. <br />
** Up to differential equations on chaotic systems. <br />
** As fine granularity of prediction increases, number of factors and density of data quickly becomes too much for most humans to consider <br />
* '''Data Driven Models''' <br />
** Use ML to provide '''iterative''' gain to reduce error <br />
*** Known data→model creation→point toward new factors.<br />
** Uses a split in training and testing data, or sum of error to move toward the correct answer.<br />
** Human researchers more free to find more data, improve prediction, develop theories<br />
<br />
== <span>Techniques</span> ==<br />
<br />
=== Machine Learning in General ===<br />
'''Supervised Learning:''' the model estimates, error verifies <br />
<br />
* If incorrect, needs user input for correction.<br />
* ''Example'': a computer vision system trained to find features in images via user annotated images. <br />
<br />
'''Unsupervised Learning:''' Clustering/Grouping of similar items<br />
<br />
* Need a similarity measure via feature vectors and ability to adjust weights <br />
* ''Example'': Taste prediction algorithms used in web advertising. <br />
<br />
'''Reinforced Learning''' Model estimates a sequence of guesses <br />
<br />
* Correct if and only if the entire sequence or a parameterized output scoring<br />
* Instant feedback but high compute cost <br />
* ''Example'': Game-play in actor-environment model<br />
<br />
=== Types of Problems and Output ===<br />
'''Numeric''' function maps <math display="inline">X</math> to <math display="inline">Y</math> and output is in <math display="inline">\mathbb{R}</math>. <br />
<br />
'''Categorization''' non-orderable sorting <br />
<br />
'''Clustering''' finding principle ways groups differ <br />
<br />
'''Anomaly Detection''' finding data points which are out of the ordinary <br />
<br />
'''Actor Models''' real-time decision making or detailed simulation<br />
<br />
=== Predictive Models ===<br />
Predictive ML models which are also Linear: <br />
<br />
* Utilize a split of training and test data: test-training or <math display="inline">k</math>-fold cross-validation <br />
* use a function mapping of one or more independent variables to the dependent variables, then re-evaluate to reduce error Includes techniques for mixed model reduction<br />
* Reduction of the number of predictors via '''Lasso''', '''Ridge''', and '''Elastic Net''' techniques<br />
<br />
=== Feature-vector based models ===<br />
Nested <code>if</code> statements try to find decision boundaries by distance between independent data and dependent outcome features have weighted probability most information by Bayesian inference<br />
<br />
* '''Decision Trees''' can be used to create decision models for linearly separable data effectively a neural network with one neuron<br />
* '''Random Forest''' utilizes a number of differently-tuned trees trees provide consensus voting-based approach for non-linearly separable data smaller tree depth typically prevents over-fit<br />
<br />
=== Clustering ===<br />
Groups data into cluster such that distance within clusters is small, and between differing groups is large <br />
<br />
Works with any well-defined &quot;distance&quot; function: Euclidean, Hamming, Inner Product, etc. <br />
<br />
'''k-Means Clustering:''' <br />
<br />
* choose <math display="inline">k</math> number of clusters randomly distribute <math display="inline">k</math> points, '''centroids''', into feature space <br />
* divide and classify data by distance to <math display="inline">k</math> centroids <br />
* move centroids based on center of groups repeats until convergence to some epsilon value<br />
* where points no longer move across iterations <br />
* '''Goodness of Fit''' for <math display="inline">N-M</math> possible values of <math display="inline">k</math>, an inflection in overall likelihood ratio given by probability function for set<br />
<br />
'''K-nearest Neighbors:''' <br />
<br />
Creates a probabilistic decision boundary within a feature space between <math display="inline">K</math> centroids<br />
<br />
Unsupervised system to find structures of data works on majority voting system<br />
<br />
'''Gradient Boosting:'''<br />
<br />
System attempts to find the direction and vector of change in a dimensional field, and follow these iteratively to find local extrema. <br />
<br />
Most use some '''Quasi-Newton Method''' for finding extrema for faster centroid convergence. <br />
<br />
'''Support Vector Machine:''' <br />
<br />
Also known as SVM, applies a classifier into high-dimensional data to split points into groups some use a '''kernel trick.'''<br />
<br />
Generates the inner product space of two arbitrary-dimensional numeric matrix spaces, showing the shape of data<br />
<br />
=== Genetic Algorithms ===<br />
Utilize some adversarial scoring method of initially randomized vectors:<br />
<br />
* ’survivors’ become the basis of new models similar iterative concept to gradient methods.<br />
* Does not have to understand topology of space requires creator to specify scoring for the machine .<br />
<br />
'''Convolutional Neural Networks:'''<br />
<br />
Utilize iterative scoring between training and testing, along with '''gradient descent''' on a number of layered, weighted vectors to extract features from a complex data set. <br />
<br />
'''Vision Systems''' <br />
<br />
Take high throughput data and simplify before work is done ''ex'': 1080p @ 60fps <math display="inline">\rightarrow</math> 240x360 @ 15fps, broken into component channels Gaussian blur filter kernel applied to high-res images Edge detection via double threshold ML or CNN used past this to determine actual features<br />
</div><br />
<br />
== Software Toolkit ==<br />
<div class="outline"><br />
=== Python ===<br />
General purpose programming language with many libraries Interpreted language: each line is run one at a time by a virtual machine.<br />
<br />
'''Dependency Structure''' <br />
<br />
''system'' vs ''user'' python <br />
<br />
virtualenv: <code>$ python3 -m venv /path/to/new/environment</code> <br />
<br />
pip libraries <code>$ pip list outdated format=freeze | grep -v | cut -d=" " -f1 | xargs -n1 pip install -U</code><br />
<br />
==== Anaconda ====<br />
separate virtualenv system specifically for data science:<br />
<br />
* '''Spyder''' IDE with visual output <br />
* '''JupyterLabs''' notes with data visualizations <br />
* '''Orange''' visual IDE for stats exploration<br />
* <code>pandas, NumPy, and SciPy</code> libraries for data serialization and numerical work in python <br />
* <code>matplotlib</code> for data visualization <br />
* <code>scikit-learn</code> main library for ML <br />
* <code>DASK</code> distributed abstraction layer with <code>pandas</code> grammar to easily distribute python tasks into 1-1000 compute nodes <br />
* <code>PyTorch, and TensorFlow</code> deep learning and CNN generation systems massive compute overhead to train models require data map reduction and or imputation to run well</div><div class="outline"><br />
=== R language ===<br />
Statistical programming language: interpreter invokes compiled C or FORTRAN.<br />
<br />
Also works within Jupyter notebook for instant visualization, if wanted.<br />
<br />
Open-source and extended by the Comprehensive R Archive Network (CRAN), which includes extensive documentation.<br />
<br />
* <code>rmarkdown</code> format a document from R with optional LaTeXbindings <br />
* <code>tidyverse</code> <br />
** <code>dplyr</code> grammar for mass data manipulation <br />
** <code>ggplot2</code> a library for creating graphs and visualizations <br />
* <code>doparallel</code> cost-free abstraction, pooling of CPU threads <br />
* <code>mlr</code> interface to a large number of classification and regression techniques <br />
* <code>shiny</code> provides ability to create web servers similar to NodeJS or Python Flaskl<br />
<br />
=== Intel MKL (Math Kernel Library) ===<br />
Improves performance for Fast Fourier Transforms, linear algebra operations, vector math, deep neural networks, and kernel solvers. <br />
<br />
Default math backend for NumPy, SciPy, and MATLAB <br />
<br />
Not hardware agnostic: chooses slowest solvers for non-Intel chips by default <br />
<br />
=== OpenBLAS and LAPACK ===<br />
'''LAPACK''' (Linear Algebra PACKage) provides APIs much like MKL <br />
<br />
'''OpenBLAS''' (Basic Linear Algebra Subprograms) extends LAPACK with optimizations for parallel computing <br />
<br />
Default for R and Biopython<br />
<br />
=== Message Passing Interface (MPI) ===<br />
Supported by all major compilers (Intel and OpenMP implementations) <br />
<br />
An API supporting shared-memory multiprocessing provides backend for many parallel computing systems, allowing for multi-threaded access <br />
<br />
''ex'': <code>mpirun -np $NUM_PROC /path/to/coolProgram &lt; $INPUT &gt; /path/to/output</code> <br />
<br />
=== GNU Parallel ===<br />
simple vectorization of loops over processors for non-multithreaded processes <br />
<br />
''ex'': <code>parallel -j $NUM_PROC /path/to/thescript.sh ::: </code><span><code>1..n</code></span><code> ::: </code><span><code>1..m</code></span><br />
<br />
=== CUDA/OpenCL ===<br />
Nvidia-specific '''CUDA''' and open-source '''OpenCL''' provide a hardware abstracting API for using GPU for compute tasks <br />
<br />
must-have for Pytorch or TensorFlow workloads <br />
<br />
Nomenclature Divergence <br />
<br />
* CUDA thread = OpenCL work item = CPU lane <br />
* CUDA multiprocessor = OpenCL compute unit = CPU<br />
<br />
=== High Performance Computers ===<br />
HPC or super-computing clusters provide high throughput analysis.<br />
<br />
Amazingly high amount of computational power.<br />
<br />
Need to plan your analysis.<br />
<br />
'''NDSU Center for Computationally-Assisted Science and Technology (CCAST)''' provides a platform for these workloads connect via <code>ssh</code> uses loadable modules: <br />
<br />
* ''ex:'' <code>module load parallel</code> <br />
<br />
batch processing via PBS scripting<br />
</div><br />
<br />
== <span>Continued Learning</span> ==<br />
<div class="outline"><br />
=== General Programming/Computers Websites ===<br />
<br />
* StackOverflow.com - check before asking new questions <br />
* RosettaCode.org - data structures and algorithms in many languages <br />
* Linux.die.net/man/ - the Linux manual <br />
* grymoire.com/Unix/ - more *nix CLI tutorials<br />
<br />
=== Python ===<br />
<br />
* docs.python.org/3/ - the official python documentation <br />
* docs.python.org/3/tutorial - the official tutorial <br />
* diveintopython.net - guided tutorial online <br />
* pythontutor.com - visual debugger <br />
<br />
=== R ===<br />
<br />
* cran.r-project.org - CRAN <br />
* cran.r-project.org/manuals.html <br />
* rdrr.io - meta-manual lookup and many other tools for R <br />
* swirlstats.com - learn R, in R <br />
* statslearning.com - statistical machine learning coursework<br />
<br />
=== Recommended Reading ===<br />
<br />
* ''An Introduction to Statistical Machine Learning'' by Gareth James et al. <br />
* ''A Primer on Scientific Programming with Python'' by Hans Petter Langtangen <br />
* ''R for Data Science'' by Wickham and Grolmund</div></div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Statistical_Machine_Learning&diff=142Statistical Machine Learning2022-11-01T21:04:05Z<p>Sysadmin: </p>
<hr />
<div>=== <span>Overview</span> ===<br />
<div class="outline"><br />
A crash course to enable gentle introduction into the machine learning techniques and its applications into data science. <br />
<br />
Topics Covered: <br />
<br />
* Basic Concepts <br />
* Data Models <br />
* Machine Learning <br />
* ML Techniques <br />
* SoftwareToolkit <br />
* Continued Learning<br />
<br />
== <span>Basic Concepts</span> ==<br />
<div class="outline"><br />
=== Definitions: ===<br />
<br />
* '''Statistical Machine Learning''' is a set of tools used to model and understand complex data sets <br />
* '''Data Science''' is a set of techniques in computing to support the analysis of data <br />
** Not very useful without some domain knowledge: it is important to ''know your data''. <br />
* Includes analytic techniques: <br />
** descriptive statistics <br />
** data visualization <br />
** statistical machine learning <br />
** neural networks<br />
** actor-environment models <br />
* Also includes computational techniques: <br />
** database administration <br />
** management of information systems <br />
** parallelization <math display="inline">\rightarrow</math> high performance computing<br />
</div><br />
<br />
=== Basic Concepts ===<br />
<div class="outline"><br />
==== Knowing your data ====<br />
Technical definition: <br />
<br />
Let <math display="inline">n</math> represent a number of distinct '''observations''', and let <math display="inline">p</math> represent a number of '''predictors''' Then, our observed data <math display="inline">\textbf{X}</math> is an <math display="inline">n\times p</math> matrix with row observation vectors <math display="inline">\vec{x}_{1..n}</math> and column predictor vectors <math display="inline">\vec{x}_{1..p}</math>. <br />
<br />
In addition, we will also have '''response''' variable(s) <math display="inline">\textbf{Y}</math>, which is a made up of some <math display="inline">n</math>-length vectors. <br />
<br />
So, our combined dataset consists of <math>[(\vec{x}_1,\vec{y}_1),(\vec{x}_2, \vec{y}_2),...,(\vec{x}_n,\vec{y}_n)]</math><br />
<br />
'''Our Mission''': determine relationships between <math display="inline">\textbf{X}</math> and <math display="inline">\textbf{Y}</math> which are mathematically sound, leading to better understandin<br />
<br />
Typically a table has columns as features, rows as entries<br />
<br />
'''Entries''' might be '''numeric''' or '''categorical'''. <br />
<br />
Data sources are either '''Structured''' or '''Unstructured''': <br />
<br />
* Unstructured data will require some transformation. <br />
<br />
Some data may also be '''time series''' taking a sampling of points over time, contributing to a 3-dimensional '''Data Cub'''<br />
<br />
Several techniques can be used to reduce complex data: <br />
<br />
* '''numeric representation''' mapping of categorical information into numbers. <br />
* '''scaling''' redefine a new range for a predictor vector. <br />
* '''normalization''' redefine a predictor by its mean and standard deviation, giving a normal distribution of values. <br />
* '''dimension reduction''' lose fine grain of data, but gain understandability. <br />
* '''feature extraction''' a data mining technique in which we can generate new predictors from known information<br />
<br />
==== Modeling ====<br />
What is a model?<br />
<br />
'''Very well-known model''': Gravity is a functional model between masses, distances, and force. <math display="block">F = G\frac{m_1m_2}{r^2} \rightarrow g = \frac{G M}{r^2} \rightarrow v(t) = v(0) - gt.</math> '''<math display="block">k = Ae^{\frac{E_a}{K_b T}}.</math>Statistics definition''': <br />
<br />
Let <math display="inline">X = (\vec{x}_1, \vec{x}_2, ..., \vec{x}_p)</math> each of length <math display="inline">n</math>, and <math display="inline">Y = (\vec{y})</math> of length <math display="inline">n</math>, then for <math display="inline">X</math> and <math display="inline">Y</math>, there exists a function with a '''systematic''' <math display="inline">f</math> and '''error term''' <math display="inline">\varepsilon</math>: <math display="block">Y = f(X) + \varepsilon</math>Why do we even estimate <math display="inline">f</math> at all? '''Prediction''' or '''Inference'''<br />
<br />
'''Predictive models''' create an estimator <math display="inline">\hat{f}</math> which we can use to estimate <math display="inline">Y</math> using a sample <math display="inline">X</math> from a larger population: <math display="block">\hat{ Y} = \hat{f}(X)</math> With error: <math display="inline">E(Y - \hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \varepsilon</math>. <br />
<br />
'''Inference models''' are primarily interested in how <math display="inline">Y</math> is affected by <math display="inline">X</math>: <br />
<br />
* What predictors associated with response? <br />
* What is the relationship of predictors to response? <br />
* What is the overall nature of relationship between <math display="inline">Y</math> and the predictors.<br />
<br />
==== Signal vs Noise ====<br />
Consider '''precision''' and '''accuracy'''. <br />
<br />
* Both contribute into data set <br />
<br />
'''noise:''' variation in data which detracts from constructing '''information''', as opposed to '''signal'''–data which is representative of a system under study and contains information. <br />
<br />
High signal to noise allows us to minimize '''reducible error''', caused by sampling technique. <br />
<br />
Different than '''irreducible error''', created by factors we are not measuring.<br />
<br />
==== Error and Fit ====<br />
In Modeling In the terms of modeling, precision of a model is referred to '''variance''' and the accuracy of a model its degree of '''bias'''. <br />
<br />
Generally, overly complex models generate high variance, and can '''over-fit''' to input data, making the model useless to new data.<br />
<br />
Generally, '''Mean Square Error''' or MSE, used to determine goodness-of-fit for model calibration: <math display="inline">MSE = \frac{1}{n}\sum^n_{i=1}(y_i - \hat{f}(x_i))^2</math>. <br />
<br />
Error rate used in classification: <math display="inline">\frac{1}{n} \sum^n_{i=1} I(y_i \ne \hat{y}_i)</math>. <br />
<br />
However for reporting, <math display="inline">R^2</math> statistic is more often used, because it gives a value between 0 and 1 useful to determine how much of variance in <math display="inline">Y</math> is explained by variance in <math display="inline">X</math>: <math display="inline">R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2} = \frac{RSS}{TSS}</math><br />
<br />
Example functions relating <math display="inline">X</math> and <math display="inline">Y</math>:<br />
<br />
* A linear function: <math display="inline">Y = \beta_0 + \beta_1 X + \varepsilon</math> <br />
* A polynomial: <math display="inline">Y = \beta_0 + \beta_1 X + \beta_2 X + ... + \varepsilon</math> <br />
* A natural function: <math display="inline">Y = e^{-\alpha_0 X^{\alpha_1}} + \varepsilon</math> <br />
* A logistical function: <math display="inline">p = \frac{ e^{\beta_0 + \beta_1 X} }{1 + e^{\beta_0 + \beta_1 X} }</math> <br />
* A series of nested <code>if</code> statements <br />
* A series of differential equations: <math display="inline">x' = x_n - \bar{x}_n: y' = y_n - \beta_n x_n'</math><br />
<br />
==== Why Machine Learning? ====<br />
Types of Questions: <br />
<br />
* '''Exact Solution is known''' normal coding problems, linear models, and classical statistics. <br />
* '''Exact Solution is unknown, but can be extracted with work''' work with systems experts and domain knowledge to create code. <br />
* '''Exact Solution is known, but not yet conveyable''' ML is useful. <br />
* '''Exact Solution not known by humans''' ML and/or Deep Learning needed<br />
<br />
''Example:'' consider a prediction of temperature: <br />
<br />
* '''Knowledge Based Models''' <br />
** Physics and atmospheric science based model. <br />
** Up to differential equations on chaotic systems. <br />
** As fine granularity of prediction increases, number of factors and density of data quickly becomes too much for most humans to consider <br />
* '''Data Driven Models''' <br />
** Use ML to provide '''iterative''' gain to reduce error <br />
*** Known data→model creation→point toward new factors.<br />
** Uses a split in training and testing data, or sum of error to move toward the correct answer.<br />
** Human researchers more free to find more data, improve prediction, develop theories<br />
<br />
== <span>Techniques</span> ==<br />
<br />
=== Machine Learning in General ===<br />
'''Supervised Learning:''' the model estimates, error verifies <br />
<br />
* If incorrect, needs user input for correction.<br />
* ''Example'': a computer vision system trained to find features in images via user annotated images. <br />
<br />
'''Unsupervised Learning:''' Clustering/Grouping of similar items<br />
<br />
* Need a similarity measure via feature vectors and ability to adjust weights <br />
* ''Example'': Taste prediction algorithms used in web advertising. <br />
<br />
'''Reinforced Learning''' Model estimates a sequence of guesses <br />
<br />
* Correct if and only if the entire sequence or a parameterized output scoring<br />
* Instant feedback but high compute cost <br />
* ''Example'': Game-play in actor-environment model<br />
<br />
=== Types of Problems and Output ===<br />
'''Numeric''' function maps <math display="inline">X</math> to <math display="inline">Y</math> and output is in <math display="inline">\mathbb{R}</math>. <br />
<br />
'''Categorization''' non-orderable sorting <br />
<br />
'''Clustering''' finding principle ways groups differ <br />
<br />
'''Anomaly Detection''' finding data points which are out of the ordinary <br />
<br />
'''Actor Models''' real-time decision making or detailed simulation<br />
<br />
=== Predictive Models ===<br />
Predictive ML models which are also Linear: <br />
<br />
* Utilize a split of training and test data: test-training or <math display="inline">k</math>-fold cross-validation <br />
* use a function mapping of one or more independent variables to the dependent variables, then re-evaluate to reduce error Includes techniques for mixed model reduction<br />
* Reduction of the number of predictors via '''Lasso''', '''Ridge''', and '''Elastic Net''' techniques<br />
<br />
=== Feature-vector based models ===<br />
Nested <code>if</code> statements try to find decision boundaries by distance between independent data and dependent outcome features have weighted probability most information by Bayesian inference<br />
<br />
* '''Decision Trees''' can be used to create decision models for linearly separable data effectively a neural network with one neuron<br />
* '''Random Forest''' utilizes a number of differently-tuned trees trees provide consensus voting-based approach for non-linearly separable data smaller tree depth typically prevents over-fit<br />
<br />
=== Clustering ===<br />
Groups data into cluster such that distance within clusters is small, and between differing groups is large <br />
<br />
Works with any well-defined &quot;distance&quot; function: Euclidean, Hamming, Inner Product, etc. <br />
<br />
'''k-Means Clustering:''' <br />
<br />
* choose <math display="inline">k</math> number of clusters randomly distribute <math display="inline">k</math> points, '''centroids''', into feature space <br />
* divide and classify data by distance to <math display="inline">k</math> centroids <br />
* move centroids based on center of groups repeats until convergence to some epsilon value<br />
* where points no longer move across iterations <br />
* '''Goodness of Fit''' for <math display="inline">N-M</math> possible values of <math display="inline">k</math>, an inflection in overall likelihood ratio given by probability function for set<br />
<br />
'''K-nearest Neighbors:''' <br />
<br />
Creates a probabilistic decision boundary within a feature space between <math display="inline">K</math> centroids<br />
<br />
Unsupervised system to find structures of data works on majority voting system<br />
<br />
'''Gradient Boosting:'''<br />
<br />
System attempts to find the direction and vector of change in a dimensional field, and follow these iteratively to find local extrema. <br />
<br />
Most use some '''Quasi-Newton Method''' for finding extrema for faster centroid convergence. <br />
<br />
'''Support Vector Machine:''' <br />
<br />
Also known as SVM, applies a classifier into high-dimensional data to split points into groups some use a '''kernel trick.'''<br />
<br />
Generates the inner product space of two arbitrary-dimensional numeric matrix spaces, showing the shape of data<br />
<br />
=== '''Genetic Algorithms''' ===<br />
Utilize some adversarial scoring method of initially randomized vectors:<br />
<br />
* ’survivors’ become the basis of new models similar iterative concept to gradient methods.<br />
* Does not have to understand topology of space requires creator to specify scoring for the machine .<br />
<br />
'''Convolutional Neural Networks:'''<br />
<br />
Utilize iterative scoring between training and testing, along with '''gradient descent''' on a number of layered, weighted vectors to extract features from a complex data set. <br />
<br />
'''Vision Systems''' <br />
<br />
Take high throughput data and simplify before work is done ''ex'': 1080p @ 60fps <math display="inline">\rightarrow</math> 240x360 @ 15fps, broken into component channels Gaussian blur filter kernel applied to high-res images Edge detection via double threshold ML or CNN used past this to determine actual features<br />
</div><br />
<br />
== Software Toolkit ==<br />
<div class="outline"><br />
=== Python ===<br />
General purpose programming language with many libraries Interpreted language: each line is run one at a time by a virtual machine.<br />
<br />
'''Dependency Structure''' <br />
<br />
''system'' vs ''user'' python <br />
<br />
virtualenv: <code>$ python3 -m venv /path/to/new/environment</code> <br />
<br />
pip libraries <code>$ pip list outdated format=freeze | grep -v | cut -d=" " -f1 | xargs -n1 pip install -U</code><br />
<br />
==== '''Anaconda''' ====<br />
separate virtualenv system specifically for data science:<br />
<br />
* '''Spyder''' IDE with visual output <br />
* '''JupyterLabs''' notes with data visualizations <br />
* '''Orange''' visual IDE for stats exploration<br />
* <code>pandas, NumPy, and SciPy</code> libraries for data serialization and numerical work in python <br />
* <code>matplotlib</code> for data visualization <br />
* <code>scikit-learn</code> main library for ML <br />
* <code>DASK</code> distributed abstraction layer with <code>pandas</code> grammar to easily distribute python tasks into 1-1000 compute nodes <br />
* <code>PyTorch, and TensorFlow</code> deep learning and CNN generation systems massive compute overhead to train models require data map reduction and or imputation to run well</div><div class="outline"><br />
=== R language ===<br />
Statistical programming language: interpreter invokes compiled C or FORTRAN.<br />
<br />
Also works within Jupyter notebook for instant visualization, if wanted.<br />
<br />
Open-source and extended by the Comprehensive R Archive Network (CRAN), which includes extensive documentation.<br />
<br />
* <code>rmarkdown</code> format a document from R with optional LaTeXbindings <br />
* <code>tidyverse</code> <br />
** <code>dplyr</code> grammar for mass data manipulation <br />
** <code>ggplot2</code> a library for creating graphs and visualizations <br />
* <code>doparallel</code> cost-free abstraction, pooling of CPU threads <br />
* <code>mlr</code> interface to a large number of classification and regression techniques <br />
* <code>shiny</code> provides ability to create web servers similar to NodeJS or Python Flaskl<br />
<br />
=== Intel MKL (Math Kernel Library) ===<br />
Improves performance for Fast Fourier Transforms, linear algebra operations, vector math, deep neural networks, and kernel solvers. <br />
<br />
Default math backend for NumPy, SciPy, and MATLAB <br />
<br />
Not hardware agnostic: chooses slowest solvers for non-Intel chips by default <br />
<br />
=== OpenBLAS and LAPACK ===<br />
'''LAPACK''' (Linear Algebra PACKage) provides APIs much like MKL <br />
<br />
'''OpenBLAS''' (Basic Linear Algebra Subprograms) extends LAPACK with optimizations for parallel computing <br />
<br />
Default for R and Biopython<br />
<br />
=== Message Passing Interface (MPI) ===<br />
Supported by all major compilers (Intel and OpenMP implementations) <br />
<br />
An API supporting shared-memory multiprocessing provides backend for many parallel computing systems, allowing for multi-threaded access <br />
<br />
''ex'': <code>mpirun -np $NUM_PROC /path/to/coolProgram &lt; $INPUT &gt; /path/to/output</code> <br />
<br />
=== GNU Parallel ===<br />
simple vectorization of loops over processors for non-multithreaded processes <br />
<br />
''ex'': <code>parallel -j $NUM_PROC /path/to/thescript.sh ::: </code><span><code>1..n</code></span><code> ::: </code><span><code>1..m</code></span><br />
<br />
=== CUDA/OpenCL ===<br />
Nvidia-specific '''CUDA''' and open-source '''OpenCL''' provide a hardware abstracting API for using GPU for compute tasks <br />
<br />
must-have for Pytorch or TensorFlow workloads <br />
<br />
Nomenclature Divergence <br />
<br />
* CUDA thread = OpenCL work item = CPU lane <br />
* CUDA multiprocessor = OpenCL compute unit = CPU<br />
<br />
=== High Performance Computers ===<br />
HPC or supercomputing clusters provide high throughput analysis.<br />
<br />
Amazingly high amount of computational power.<br />
<br />
Need to plan your analysis.<br />
<br />
'''NDSU Center for Computationally-Assisted Science and Technology (CCAST)''' provides a platform for these workloads connect via <code>ssh</code> uses loadable modules: <br />
<br />
* ''ex:'' <code>module load parallel</code> <br />
<br />
batch processing via PBS scripting<br />
</div><br />
<br />
== <span>Continued Learning</span> ==<br />
<div class="outline"><br />
=== General Programming/Computers Websites ===<br />
<br />
* StackOverflow.com - check before asking new questions <br />
* RosettaCode.org - data structures and algorithms in many languages <br />
* Linux.die.net/man/ - the Linux manual <br />
* grymoire.com/Unix/ - more *nix CLI tutorials<br />
<br />
=== Python ===<br />
<br />
* docs.python.org/3/ - the official python documentation <br />
* docs.python.org/3/tutorial - the official tutorial <br />
* diveintopython.net - guided tutorial online <br />
* pythontutor.com - visual debugger <br />
<br />
=== R ===<br />
<br />
* cran.r-project.org - CRAN <br />
* cran.r-project.org/manuals.html <br />
* rdrr.io - meta-manual lookup and many other tools for R <br />
* swirlstats.com - learn R, in R <br />
* statlearning.com - statistical machine learning coursework<br />
<br />
=== Recommended Reading ===<br />
<br />
* ''An Introduction to Statistical Machine Learning'' by Gareth James et al. <br />
* ''A Primer on Scientific Programming with Python'' by Hans Petter Langtangen <br />
* ''R for Data Science'' by Wickham and Grolmund</div></div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Statistical_Machine_Learning&diff=141Statistical Machine Learning2022-10-21T20:41:25Z<p>Sysadmin: </p>
<hr />
<div>=== <span>Overview</span> ===<br />
<div class="outline"><br />
A crash course to enable gentle introduction into the machine learning techniques and its applications into data science. <br />
<br />
Topics Covered: <br />
<br />
* Basic Concepts <br />
* Data Models <br />
* Machine Learning <br />
* ML Techniques <br />
* SoftwareToolkit <br />
* Continued Learning<br />
<br />
== <span>Basic Concepts</span> ==<br />
<div class="outline"><br />
=== Definitions: ===<br />
<br />
* '''Statistical Machine Learning''' is a set of tools used to model and understand complex data sets <br />
* '''Data Science''' is a set of techniques in computing to support the analysis of data <br />
** Not very useful without some domain knowledge: it is important to ''know your data''. <br />
* Includes analytic techniques: <br />
** descriptive statistics <br />
** data visualization <br />
** statistical machine learning <br />
** neural networks<br />
** actor-environment models <br />
* Also includes computational techniques: <br />
** database administration <br />
** management of information systems <br />
** parallelization <math display="inline">\rightarrow</math> high performance computing<br />
</div><br />
<br />
=== Basic Concepts ===<br />
<div class="outline"><br />
==== Knowing your data ====<br />
Technical definition: <br />
<br />
Let <math display="inline">n</math> represent a number of distinct '''observations''', and let <math display="inline">p</math> represent a number of '''predictors''' Then, our observed data <math display="inline">\textbf{X}</math> is an <math display="inline">n\times p</math> matrix with row observation vectors <math display="inline">\vec{x}_{1..n}</math> and column predictor vectors <math display="inline">\vec{x}_{1..p}</math>. <br />
<br />
In addition, we will also have '''response''' variable(s) <math display="inline">\textbf{Y}</math>, which is a made up of some <math display="inline">n</math>-length vectors. <br />
<br />
So, our combined dataset consists of <math>[(\vec{x}_1,\vec{y}_1),(\vec{x}_2, \vec{y}_2),...,(\vec{x}_n,\vec{y}_n)]</math><br />
<br />
'''Our Mission''': determine relationships between <math display="inline">\textbf{X}</math> and <math display="inline">\textbf{Y}</math> which are mathematically sound, leading to better understandin<br />
<br />
Typically a table has columns as features, rows as entries<br />
<br />
'''Entries''' might be '''numeric''' or '''categorical'''. <br />
<br />
Data sources are either '''Structured''' or '''Unstructured''': <br />
<br />
* Unstructured data will require some transformation. <br />
<br />
Some data may also be '''time series''' taking a sampling of points over time, contributing to a 3-dimensional '''Data Cub'''<br />
<br />
Several techniques can be used to reduce complex data: <br />
<br />
* '''numeric representation''' mapping of categorical information into numbers. <br />
* '''scaling''' redefine a new range for a predictor vector. <br />
* '''normalization''' redefine a predictor by its mean and standard deviation, giving a normal distribution of values. <br />
* '''dimension reduction''' lose fine grain of data, but gain understandability. <br />
* '''feature extraction''' a data mining technique in which we can generate new predictors from known information<br />
<br />
==== Modeling ====<br />
What is a model?<br />
<br />
'''Very well-known model''': Gravity is a functional model between masses, distances, and force. <math display="block">F = G\frac{m_1m_2}{r^2} \rightarrow g = \frac{G M}{r^2} \rightarrow v(t) = v(0) - gt.</math> '''<math display="block">k = Ae^{\frac{E_a}{K_b T}}.</math>Statistics definition''': <br />
<br />
Let <math display="inline">X = (\vec{x}_1, \vec{x}_2, ..., \vec{x}_p)</math> each of length <math display="inline">n</math>, and <math display="inline">Y = (\vec{y})</math> of length <math display="inline">n</math>, then for <math display="inline">X</math> and <math display="inline">Y</math>, there exists a function with a '''systematic''' <math display="inline">f</math> and '''error term''' <math display="inline">\varepsilon</math>: <math display="block">Y = f(X) + \varepsilon</math>Why do we even estimate <math display="inline">f</math> at all? '''Prediction''' or '''Inference'''<br />
<br />
'''Predictive models''' create an estimator <math display="inline">\hat{f}</math> which we can use to estimate <math display="inline">Y</math> using a sample <math display="inline">X</math> from a larger population: <math display="block">\hat{ Y} = \hat{f}(X)</math> With error: <math display="inline">E(Y - \hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \varepsilon</math>. <br />
<br />
'''Inference models''' are primarily interested in how <math display="inline">Y</math> is affected by <math display="inline">X</math>: <br />
<br />
* What predictors associated with response? <br />
* What is the relationship of predictors to response? <br />
* What is the overall nature of relationship between <math display="inline">Y</math> and the predictors.<br />
<br />
==== Signal vs Noise ====<br />
Consider '''precision''' and '''accuracy'''. <br />
<br />
* Both contribute into data set <br />
<br />
'''noise:''' variation in data which detracts from constructing '''information''', as opposed to '''signal'''–data which is representative of a system under study and contains information. <br />
<br />
High signal to noise allows us to minimize '''reducible error''', caused by sampling technique. <br />
<br />
Different than '''irreducible error''', created by factors we are not measuring.<br />
<br />
==== Error and Fit ====<br />
In Modeling In the terms of modeling, precision of a model is referred to '''variance''' and the accuracy of a model its degree of '''bias'''. <br />
<br />
Generally, overly complex models generate high variance, and can '''over-fit''' to input data, making the model useless to new data.<br />
<br />
Generally, '''Mean Square Error''' or MSE, used to determine goodness-of-fit for model calibration: <math display="inline">MSE = \frac{1}{n}\sum^n_{i=1}(y_i - \hat{f}(x_i))^2</math>. <br />
<br />
Error rate used in classification: <math display="inline">\frac{1}{n} \sum^n_{i=1} I(y_i \ne \hat{y}_i)</math>. <br />
<br />
However for reporting, <math display="inline">R^2</math> statistic is more often used, because it gives a value between 0 and 1 useful to determine how much of variance in <math display="inline">Y</math> is explained by variance in <math display="inline">X</math>: <math display="inline">R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2} = \frac{RSS}{TSS}</math><br />
<br />
Example functions relating <math display="inline">X</math> and <math display="inline">Y</math>:<br />
<br />
* A linear function: <math display="inline">Y = \beta_0 + \beta_1 X + \varepsilon</math> <br />
* A polynomial: <math display="inline">Y = \beta_0 + \beta_1 X + \beta_2 X + ... + \varepsilon</math> <br />
* A natural function: <math display="inline">Y = e^{-\alpha_0 X^{\alpha_1}} + \varepsilon</math> <br />
* A logistical function: <math display="inline">p = \frac{ e^{\beta_0 + \beta_1 X} }{1 + e^{\beta_0 + \beta_1 X} }</math> <br />
* A series of nested <code>if</code> statements <br />
* A series of differential equations: <math display="inline">x' = x_n - \bar{x}_n: y' = y_n - \beta_n x_n'</math><br />
<br />
==== Why Machine Learning? ====<br />
Types of Questions: <br />
<br />
* '''Exact Solution is known''' normal coding problems, linear models, and classical statistics. <br />
* '''Exact Solution is unknown, but can be extracted with work''' work with systems experts and domain knowledge to create code. <br />
* '''Exact Solution is known, but not yet conveyable''' ML is useful. <br />
* '''Exact Solution not known by humans''' ML and/or Deep Learning needed<br />
<br />
''Example:'' consider a prediction of temperature: <br />
<br />
* '''Knowledge Based Models''' <br />
** Physics and atmospheric science based model. <br />
** Up to differential equations on chaotic systems. <br />
** As fine granularity of prediction increases, number of factors and density of data quickly becomes too much for most humans to consider <br />
* '''Data Driven Models''' <br />
** Use ML to provide '''iterative''' gain to reduce error <br />
*** Known data→model creation→point toward new factors.<br />
** Uses a split in training and testing data, or sum of error to move toward the correct answer.<br />
** Human researchers more free to find more data, improve prediction, develop theories<br />
<br />
== <span>Techniques</span> ==<br />
<br />
=== Machine Learning in General ===<br />
'''Supervised Learning:''' the model estimates, error verifies <br />
<br />
* If incorrect, needs user input for correction.<br />
* ''Example'': a computer vision system trained to find features in images via user annotated images. <br />
<br />
'''Unsupervised Learning:''' Clustering/Grouping of similar items<br />
<br />
* Need a similarity measure via feature vectors and ability to adjust weights <br />
* ''Example'': Taste prediction algorithms used in web advertising. <br />
<br />
'''Reinforced Learning''' Model estimates a sequence of guesses <br />
<br />
* Correct if and only if the entire sequence or a parameterized output scoring<br />
* Instant feedback but high compute cost <br />
* ''Example'': Game-play in actor-environment model<br />
<br />
=== Types of Problems and Output ===<br />
'''Numeric''' function maps <math display="inline">X</math> to <math display="inline">Y</math> and output is in <math display="inline">\mathbb{R}</math>. <br />
<br />
'''Categorization''' non-orderable sorting <br />
<br />
'''Clustering''' finding principle ways groups differ <br />
<br />
'''Anomaly Detection''' finding data points which are out of the ordinary <br />
<br />
'''Actor Models''' real-time decision making or detailed simulation<br />
<br />
=== Predictive Models ===<br />
Predictive ML models which are also Linear: <br />
<br />
* Utilize a split of training and test data: test-training or <math display="inline">k</math>-fold cross-validation <br />
* use a function mapping of one or more independent variables to the dependent variables, then re-evaluate to reduce error Includes techniques for mixed model reduction<br />
* Reduction of the number of predictors via '''Lasso''', '''Ridge''', and '''Elastic Net''' techniques<br />
<br />
=== Feature-vector based models ===<br />
Nested <code>if</code> statements try to find decision boundaries by distance between independent data and dependent outcome features have weighted probability most information by Bayesian inference<br />
<br />
* '''Decision Trees''' can be used to create decision models for linearly separable data effectively a neural network with one neuron<br />
* '''Random Forest''' utilizes a number of differently-tuned trees trees provide consensus voting-based approach for non-linearly separable data smaller tree depth typically prevents over-fit<br />
<br />
=== Clustering ===<br />
Groups data into cluster such that distance within clusters is small, and between differing groups is large <br />
<br />
Works with any well-defined &quot;distance&quot; function: Euclidean, Hamming, Inner Product, etc. <br />
<br />
'''k-Means Clustering:''' <br />
<br />
* choose <math display="inline">k</math> number of clusters randomly distribute <math display="inline">k</math> points, '''centroids''', into feature space <br />
* divide and classify data by distance to <math display="inline">k</math> centroids <br />
* move centroids based on center of groups repeats until convergence to some epsilon value<br />
* where points no longer move across iterations <br />
* '''Goodness of Fit''' for <math display="inline">N-M</math> possible values of <math display="inline">k</math>, an inflection in overall likelihood ratio given by probability function for set<br />
<br />
'''K-nearest Neighbors:''' <br />
<br />
Creates a probabilistic decision boundary within a feature space between <math display="inline">K</math> centroids<br />
<br />
Unsupervised system to find structures of data works on majority voting system<br />
<br />
'''Gradient Boosting:'''<br />
<br />
System attempts to find the direction and vector of change in a dimensional field, and follow these iteratively to find local extrema. <br />
<br />
Most use some '''Quasi-Newton Method''' for finding extrema for faster centroid convergence. <br />
<br />
'''Support Vector Machine:''' <br />
<br />
Also known as SVM, applies a classifier into high-dimensional data to split points into groups some use a '''kernel trick.'''<br />
<br />
Generates the inner product space of two arbitrary-dimensional numeric matrix spaces, showing the shape of data<br />
<br />
=== '''Genetic Algorithms''' ===<br />
Utilize some adversarial scoring method of initially randomized vectors:<br />
<br />
* ’survivors’ become the basis of new models similar iterative concept to gradient methods.<br />
* Does not have to understand topology of space requires creator to specify scoring for the machine .<br />
<br />
'''Convolutional Neural Networks:'''<br />
<br />
Utilize iterative scoring between training and testing, along with '''gradient descent''' on a number of layered, weighted vectors to extract features from a complex data set. <br />
<br />
'''Vision Systems''' <br />
<br />
Take high throughput data and simplify before work is done ''ex'': 1080p @ 60fps <math display="inline">\rightarrow</math> 240x360 @ 15fps, broken into component channels Gaussian blur filter kernel applied to high-res images Edge detection via double threshold ML or CNN used past this to determine actual features<br />
</div><br />
<br />
== Software Toolkit ==<br />
<div class="outline"><br />
=== Python ===<br />
General purpose programming language with many libraries Interpreted language: each line is run one at a time by a virtual machine.<br />
<br />
'''Dependency Structure''' <br />
<br />
''system'' vs ''user'' python <br />
<br />
virtualenv: <code>$ python3 -m venv /path/to/new/environment</code> <br />
<br />
pip libraries <code>$ pip list outdated format=freeze | grep -v | cut -d=" " -f1 | xargs -n1 pip install -U</code><br />
<br />
==== '''Anaconda''' ====<br />
separate virtualenv system specifically for data science:<br />
<br />
* '''Spyder''' IDE with visual output <br />
* '''JupyterLabs''' notes with data visualizations <br />
* '''Orange''' visual IDE for stats exploration<br />
* <code>pandas, NumPy, and SciPy</code> libraries for data serialization and numerical work in python <br />
* <code>matplotlib</code> for data visualization <br />
* <code>scikit-learn</code> main library for ML <br />
* <code>DASK</code> distributed abstraction layer with <code>pandas</code> grammar to easily distribute python tasks into 1-1000 compute nodes <br />
* <code>PyTorch, and TensorFlow</code> deep learning and CNN generation systems massive compute overhead to train models require data map reduction and or imputation to run well</div><div class="outline"><br />
=== R language ===<br />
Statistical programming language: interpreter invokes compiled C or FORTRAN.<br />
<br />
Also works within Jupyter notebook for instant visualization, if wanted.<br />
<br />
Open-source and extended by the Comprehensive R Archive Network (CRAN), which includes extensive documentation.<br />
<br />
* <code>rmarkdown</code> format a document from R with optional LaTeXbindings <br />
* <code>tidyverse</code> <br />
** <code>dplyr</code> grammar for mass data manipulation <br />
** <code>ggplot2</code> a library for creating graphs and visualizations <br />
* <code>doparallel</code> cost-free abstraction, pooling of CPU threads <br />
* <code>mlr</code> interface to a large number of classification and regression techniques <br />
* <code>shiny</code> provides ability to create web servers similar to NodeJS or Python Flaskl<br />
<br />
=== Intel MKL (Math Kernel Library) ===<br />
Improves performance for Fast Fourier Transforms, linear algebra operations, vector math, deep neural networks, and kernel solvers. <br />
<br />
Default math backend for NumPy, SciPy, and MATLAB <br />
<br />
Not hardware agnostic: chooses slowest solvers for non-Intel chips by default <br />
<br />
=== OpenBLAS and LAPACK ===<br />
'''LAPACK''' (Linear Algebra PACKage) provides APIs much like MKL <br />
<br />
'''OpenBLAS''' (Basic Linear Algebra Subprograms) extends LAPACK with optimizations for parallel computing <br />
<br />
Default for R and Biopython<br />
<br />
=== Message Passing Interface (MPI) ===<br />
Supported by all major compilers (Intel and OpenMP implementations) <br />
<br />
An API supporting shared-memory multiprocessing provides backend for many parallel computing systems, allowing for multi-threaded access <br />
<br />
''ex'': <code>mpirun -np $NUM_PROC /path/to/coolProgram &lt; $INPUT &gt; /path/to/output</code> <br />
<br />
=== GNU Parallel ===<br />
simple vectorization of loops over processors for non-multithreaded processes <br />
<br />
''ex'': <code>parallel -j $NUM_PROC /path/to/thescript.sh ::: </code><span><code>1..n</code></span><code> ::: </code><span><code>1..m</code></span><br />
<br />
=== CUDA/OpenCL ===<br />
Nvidia-specific '''CUDA''' and open-source '''OpenCL''' provide a hardware abstracting API for using GPU for compute tasks <br />
<br />
must-have for Pytorch or TensorFlow workloads <br />
<br />
Nomenclature Divergence <br />
<br />
* CUDA thread = OpenCL work item = CPU lane <br />
* CUDA multiprocessor = OpenCL compute unit = CPU<br />
<br />
=== High Performance Computers ===<br />
HPC or supercomputing clusters provide high throughput analysis.<br />
<br />
Amazingly high amount of computational power.<br />
<br />
Need to plan your analysis.<br />
<br />
'''NDSU Center for Computationally-Assisted Science and Technology (CCAST)''' provides a platform for these workloads connect via <code>ssh</code> uses loadable modules: <br />
<br />
* ''ex:'' <code>module load parallel</code> <br />
<br />
batch processing via PBS scripting<br />
</div><br />
<br />
== <span>Continued Learning</span> ==<br />
<div class="outline"><br />
=== General Programming/Computers Websites ===<br />
<br />
* StackOverflow.com - check before asking new questions <br />
* RosettaCode.org - data structures and algorithms in many languages <br />
* Linux.die.net/man/ - the Linux manual <br />
* grymoire.com/Unix/ - more *nix CLI tutorials<br />
<br />
=== Python ===<br />
<br />
* docs.python.org/3/ - the official python documentation <br />
* docs.python.org/3/tutorial - the official tutorial <br />
* diveintopython.net - guided tutorial online <br />
* pythontutor.com - visual debugger <br />
<br />
=== R ===<br />
<br />
* cran.r-project.org - CRAN <br />
* cran.r-project.org/manuals.html <br />
* rdrr.io - meta-manual lookup and many other tools for R <br />
* swirlstats.com - learn R, in R <br />
* statslearning.com - statistical machine learning coursework<br />
<br />
=== Recommended Reading ===<br />
<br />
* ''An Introduction to Statistical Machine Learning'' by Gareth James et al. <br />
* ''A Primer on Scientific Programming with Python'' by Hans Petter Langtangen <br />
* ''R for Data Science'' by Wickham and Grolmund</div></div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Statistical_Machine_Learning&diff=140Statistical Machine Learning2022-10-21T20:39:41Z<p>Sysadmin: </p>
<hr />
<div>=== <span>Overview</span> ===<br />
<div class="outline"><br />
A crash course to enable gentle introduction into the machine learning techniques and its applications into data science. <br />
<br />
Topics Covered: <br />
<br />
* Basic Concepts <br />
* Data Models <br />
* Machine Learning <br />
* ML Techniques <br />
* SoftwareToolkit <br />
* Continued Learning<br />
<br />
== <span>Basic Concepts</span> ==<br />
<div class="outline"><br />
=== Definitions: ===<br />
<br />
* '''Statistical Machine Learning''' is a set of tools used to model and understand complex data sets <br />
* '''Data Science''' is a set of techniques in computing to support the analysis of data <br />
** Not very useful without some domain knowledge: it is important to ''know your data''. <br />
* Includes analytic techniques: <br />
** descriptive statistics <br />
** data visualization <br />
** statistical machine learning <br />
** neural networks<br />
** actor-environment models <br />
* Also includes computational techniques: <br />
** database administration <br />
** management of information systems <br />
** parallelization <math display="inline">\rightarrow</math> high performance computing<br />
</div><br />
<br />
=== Basic Concepts ===<br />
<div class="outline"><br />
==== Knowing your data ====<br />
Technical definition: <br />
<br />
Let <math display="inline">n</math> represent a number of distinct '''observations''', and let <math display="inline">p</math> represent a number of '''predictors''' Then, our observed data <math display="inline">\textbf{X}</math> is an <math display="inline">n\times p</math> matrix with row observation vectors <math display="inline">\vec{x}_{1..n}</math> and column predictor vectors <math display="inline">\vec{x}_{1..p}</math>. <br />
<br />
In addition, we will also have '''response''' variable(s) <math display="inline">\textbf{Y}</math>, which is a made up of some <math display="inline">n</math>-length vectors. <br />
<br />
So, our combined dataset consists of <math>[(\vec{x}_1,\vec{y}_1),(\vec{x}_2, \vec{y}_2),...,(\vec{x}_n,\vec{y}_n)]</math><br />
<br />
'''Our Mission''': determine relationships between <math display="inline">\textbf{X}</math> and <math display="inline">\textbf{Y}</math> which are mathematically sound, leading to better understandin<br />
<br />
Typically a table has columns as features, rows as entries<br />
<br />
'''Entries''' might be '''numeric''' or '''categorical'''. <br />
<br />
Data sources are either '''Structured''' or '''Unstructured''': <br />
<br />
* Unstructured data will require some transformation. <br />
<br />
Some data may also be '''time series''' taking a sampling of points over time, contributing to a 3-dimensional '''Data Cub'''<br />
<br />
Several techniques can be used to reduce complex data: <br />
<br />
* '''numeric representation''' mapping of categorical information into numbers. <br />
* '''scaling''' redefine a new range for a predictor vector. <br />
* '''normalization''' redefine a predictor by its mean and standard deviation, giving a normal distribution of values. <br />
* '''dimension reduction''' lose fine grain of data, but gain understandability. <br />
* '''feature extraction''' a data mining technique in which we can generate new predictors from known information<br />
<br />
==== Modeling ====<br />
What is a model?<br />
<br />
'''Very well-known model''': Gravity is a functional model between masses, distances, and force. <math display="block">F = G\frac{m_1m_2}{r^2} \rightarrow g = \frac{G M}{r^2} \rightarrow v(t) = v(0) - gt.</math> '''<math display="block">k = Ae^{\frac{E_a}{K_b T}}.</math>Statistics definition''': <br />
<br />
Let <math display="inline">X = (\vec{x}_1, \vec{x}_2, ..., \vec{x}_p)</math> each of length <math display="inline">n</math>, and <math display="inline">Y = (\vec{y})</math> of length <math display="inline">n</math>, then for <math display="inline">X</math> and <math display="inline">Y</math>, there exists a function with a '''systematic''' <math display="inline">f</math> and '''error term''' <math display="inline">\varepsilon</math>: <math display="block">Y = f(X) + \varepsilon</math>Why do we even estimate <math display="inline">f</math> at all? '''Prediction''' or '''Inference'''<br />
<br />
'''Predictive models''' create an estimator <math display="inline">\hat{f}</math> which we can use to estimate <math display="inline">Y</math> using a sample <math display="inline">X</math> from a larger population: <math display="block">\hat{ Y} = \hat{f}(X)</math> With error: <math display="inline">E(Y - \hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \varepsilon</math>. <br />
<br />
'''Inference models''' are primarily interested in how <math display="inline">Y</math> is affected by <math display="inline">X</math>: <br />
<br />
* What predictors associated with response? <br />
* What is the relationship of predictors to response? <br />
* What is the overall nature of relationship between <math display="inline">Y</math> and the predictors.<br />
<br />
==== Signal vs Noise ====<br />
Consider '''precision''' and '''accuracy'''. <br />
<br />
* Both contribute into data set <br />
<br />
'''noise:''' variation in data which detracts from constructing '''information''', as opposed to '''signal'''–data which is representative of a system under study and contains information. <br />
<br />
High signal to noise allows us to minimize '''reducible error''', caused by sampling technique. <br />
<br />
Different than '''irreducible error''', created by factors we are not measuring.<br />
<br />
==== Error and Fit ====<br />
In Modeling In the terms of modeling, precision of a model is referred to '''variance''' and the accuracy of a model its degree of '''bias'''. <br />
<br />
Generally, overly complex models generate high variance, and can '''over-fit''' to input data, making the model useless to new data.<br />
<br />
Generally, '''Mean Square Error''' or MSE, used to determine goodness-of-fit for model calibration: <math display="inline">MSE = \frac{1}{n}\sum^n_{i=1}(y_i - \hat{f}(x_i))^2</math>. <br />
<br />
Error rate used in classification: <math display="inline">\frac{1}{n} \sum^n_{i=1} I(y_i \ne \hat{y}_i)</math>. <br />
<br />
However for reporting, <math display="inline">R^2</math> statistic is more often used, because it gives a value between 0 and 1 useful to determine how much of variance in <math display="inline">Y</math> is explained by variance in <math display="inline">X</math>: <math display="inline">R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2} = \frac{RSS}{TSS}</math><br />
<br />
Example functions relating <math display="inline">X</math> and <math display="inline">Y</math>:<br />
<br />
* A linear function: <math display="inline">Y = \beta_0 + \beta_1 X + \varepsilon</math> <br />
* A polynomial: <math display="inline">Y = \beta_0 + \beta_1 X + \beta_2 X + ... + \varepsilon</math> <br />
* A natural function: <math display="inline">Y = e^{-\alpha_0 X^{\alpha_1}} + \varepsilon</math> <br />
* A logistical function: <math display="inline">p = \frac{ e^{\beta_0 + \beta_1 X} }{1 + e^{\beta_0 + \beta_1 X} }</math> <br />
* A series of nested <code>if</code> statements <br />
* A series of differential equations: <math display="inline">x' = x_n - \bar{x}_n: y' = y_n - \beta_n x_n'</math><br />
<br />
==== Why Machine Learning? ====<br />
Types of Questions: <br />
<br />
* '''Exact Solution is known''' normal coding problems, linear models, and classical statistics. <br />
* '''Exact Solution is unknown, but can be extracted with work''' work with systems experts and domain knowledge to create code. <br />
* '''Exact Solution is known, but not yet conveyable''' ML is useful. <br />
* '''Exact Solution not known by humans''' ML and/or Deep Learning needed<br />
<br />
''Example:'' consider a prediction of temperature: <br />
<br />
* '''Knowledge Based Models''' <br />
** Physics and atmospheric science based model. <br />
** Up to differential equations on chaotic systems. <br />
** As fine granularity of prediction increases, number of factors and density of data quickly becomes too much for most humans to consider <br />
* '''Data Driven Models''' <br />
** Use ML to provide '''iterative''' gain to reduce error <br />
*** Known data {\textstyle \rightarrow } model creation {\textstyle \rightarrow } point toward new factors.<br />
** Uses a split in training and testing data, or sum of error to move toward the correct answer.<br />
** Human researchers more free to find more data, improve prediction, develop theories<br />
<br />
== <span>Techniques</span> ==<br />
<br />
=== Machine Learning in General ===<br />
'''Supervised Learning:''' the model estimates, error verifies <br />
<br />
* If incorrect, needs user input for correction.<br />
* ''Example'': a computer vision system trained to find features in images via user annotated images. <br />
<br />
'''Unsupervised Learning:''' Clustering/Grouping of similar items<br />
<br />
* Need a similarity measure via feature vectors and ability to adjust weights <br />
* ''Example'': Taste prediction algorithms used in web advertising. <br />
<br />
'''Reinforced Learning''' Model estimates a sequence of guesses <br />
<br />
* Correct if and only if the entire sequence or a parameterized output scoring<br />
* Instant feedback but high compute cost <br />
* ''Example'': Game-play in actor-environment model<br />
<br />
=== Types of Problems and Output ===<br />
'''Numeric''' function maps <math display="inline">X</math> to <math display="inline">Y</math> and output is in <math display="inline">\mathbb{R}</math>. <br />
<br />
'''Categorization''' non-orderable sorting <br />
<br />
'''Clustering''' finding principle ways groups differ <br />
<br />
'''Anomaly Detection''' finding data points which are out of the ordinary <br />
<br />
'''Actor Models''' real-time decision making or detailed simulation<br />
<br />
=== Predictive Models ===<br />
Predictive ML models which are also Linear: <br />
<br />
* Utilize a split of training and test data: test-training or <math display="inline">k</math>-fold cross-validation <br />
* use a function mapping of one or more independent variables to the dependent variables, then re-evaluate to reduce error Includes techniques for mixed model reduction<br />
* Reduction of the number of predictors via '''Lasso''', '''Ridge''', and '''Elastic Net''' techniques<br />
<br />
=== Feature-vector based models ===<br />
Nested <code>if</code> statements try to find decision boundaries by distance between independent data and dependent outcome features have weighted probability most information by Bayesian inference<br />
<br />
* '''Decision Trees''' can be used to create decision models for linearly separable data effectively a neural network with one neuron<br />
* '''Random Forest''' utilizes a number of differently-tuned trees trees provide consensus voting-based approach for non-linearly separable data smaller tree depth typically prevents over-fit<br />
<br />
=== Clustering ===<br />
Groups data into cluster such that distance within clusters is small, and between differing groups is large <br />
<br />
Works with any well-defined &quot;distance&quot; function: Euclidean, Hamming, Inner Product, etc. <br />
<br />
'''k-Means Clustering:''' <br />
<br />
* choose <math display="inline">k</math> number of clusters randomly distribute <math display="inline">k</math> points, '''centroids''', into feature space <br />
* divide and classify data by distance to <math display="inline">k</math> centroids <br />
* move centroids based on center of groups repeats until convergence to some epsilon value<br />
* where points no longer move across iterations <br />
* '''Goodness of Fit''' for <math display="inline">N-M</math> possible values of <math display="inline">k</math>, an inflection in overall likelihood ratio given by probability function for set<br />
<br />
'''K-nearest Neighbors:''' <br />
<br />
Creates a probabilistic decision boundary within a feature space between <math display="inline">K</math> centroids<br />
<br />
Unsupervised system to find structures of data works on majority voting system<br />
<br />
'''Gradient Boosting:'''<br />
<br />
System attempts to find the direction and vector of change in a dimensional field, and follow these iteratively to find local extrema. <br />
<br />
Most use some '''Quasi-Newton Method''' for finding extrema for faster centroid convergence. <br />
<br />
'''Support Vector Machine:''' <br />
<br />
Also known as SVM, applies a classifier into high-dimensional data to split points into groups some use a '''kernel trick.'''<br />
<br />
Generates the inner product space of two arbitrary-dimensional numeric matrix spaces, showing the shape of data<br />
<br />
=== '''Genetic Algorithms''' ===<br />
Utilize some adversarial scoring method of initially randomized vectors:<br />
<br />
* ’survivors’ become the basis of new models similar iterative concept to gradient methods.<br />
* Does not have to understand topology of space requires creator to specify scoring for the machine .<br />
<br />
'''Convolutional Neural Networks:'''<br />
<br />
Utilize iterative scoring between training and testing, along with '''gradient descent''' on a number of layered, weighted vectors to extract features from a complex data set. <br />
<br />
'''Vision Systems''' <br />
<br />
Take high throughput data and simplify before work is done ''ex'': 1080p @ 60fps <math display="inline">\rightarrow</math> 240x360 @ 15fps, broken into component channels Gaussian blur filter kernel applied to high-res images Edge detection via double threshold ML or CNN used past this to determine actual features<br />
</div><br />
<br />
== Software Toolkit ==<br />
<div class="outline"><br />
=== Python ===<br />
General purpose programming language with many libraries Interpreted language: each line is run one at a time by a virtual machine.<br />
<br />
'''Dependency Structure''' <br />
<br />
''system'' vs ''user'' python <br />
<br />
virtualenv: <code>$ python3 -m venv /path/to/new/environment</code> <br />
<br />
pip libraries <code>$ pip list outdated format=freeze | grep -v | cut -d=" " -f1 | xargs -n1 pip install -U</code><br />
<br />
==== '''Anaconda''' ====<br />
separate virtualenv system specifically for data science:<br />
<br />
* '''Spyder''' IDE with visual output <br />
* '''JupyterLabs''' notes with data visualizations <br />
* '''Orange''' visual IDE for stats exploration<br />
* <code>pandas, NumPy, and SciPy</code> libraries for data serialization and numerical work in python <br />
* <code>matplotlib</code> for data visualization <br />
* <code>scikit-learn</code> main library for ML <br />
* <code>DASK</code> distributed abstraction layer with <code>pandas</code> grammar to easily distribute python tasks into 1-1000 compute nodes <br />
* <code>PyTorch, and TensorFlow</code> deep learning and CNN generation systems massive compute overhead to train models require data map reduction and or imputation to run well</div><div class="outline"><br />
=== R language ===<br />
Statistical programming language: interpreter invokes compiled C or FORTRAN.<br />
<br />
Also works within Jupyter notebook for instant visualization, if wanted.<br />
<br />
Open-source and extended by the Comprehensive R Archive Network (CRAN), which includes extensive documentation.<br />
<br />
* <code>rmarkdown</code> format a document from R with optional LaTeXbindings <br />
* <code>tidyverse</code> <br />
** <code>dplyr</code> grammar for mass data manipulation <br />
** <code>ggplot2</code> a library for creating graphs and visualizations <br />
* <code>doparallel</code> cost-free abstraction, pooling of CPU threads <br />
* <code>mlr</code> interface to a large number of classification and regression techniques <br />
* <code>shiny</code> provides ability to create web servers similar to NodeJS or Python Flaskl<br />
<br />
=== Intel MKL (Math Kernel Library) ===<br />
Improves performance for Fast Fourier Transforms, linear algebra operations, vector math, deep neural networks, and kernel solvers. <br />
<br />
Default math backend for NumPy, SciPy, and MATLAB <br />
<br />
Not hardware agnostic: chooses slowest solvers for non-Intel chips by default <br />
<br />
=== OpenBLAS and LAPACK ===<br />
'''LAPACK''' (Linear Algebra PACKage) provides APIs much like MKL <br />
<br />
'''OpenBLAS''' (Basic Linear Algebra Subprograms) extends LAPACK with optimizations for parallel computing <br />
<br />
Default for R and Biopython<br />
<br />
=== Message Passing Interface (MPI) ===<br />
Supported by all major compilers (Intel and OpenMP implementations) <br />
<br />
An API supporting shared-memory multiprocessing provides backend for many parallel computing systems, allowing for multi-threaded access <br />
<br />
''ex'': <code>mpirun -np $NUM_PROC /path/to/coolProgram &lt; $INPUT &gt; /path/to/output</code> <br />
<br />
=== GNU Parallel ===<br />
simple vectorization of loops over processors for non-multithreaded processes <br />
<br />
''ex'': <code>parallel -j $NUM_PROC /path/to/thescript.sh ::: </code><span><code>1..n</code></span><code> ::: </code><span><code>1..m</code></span><br />
<br />
=== CUDA/OpenCL ===<br />
Nvidia-specific '''CUDA''' and open-source '''OpenCL''' provide a hardware abstracting API for using GPU for compute tasks <br />
<br />
must-have for Pytorch or TensorFlow workloads <br />
<br />
Nomenclature Divergence <br />
<br />
* CUDA thread = OpenCL work item = CPU lane <br />
* CUDA multiprocessor = OpenCL compute unit = CPU<br />
<br />
=== High Performance Computers ===<br />
HPC or supercomputing clusters provide high throughput analysis.<br />
<br />
Amazingly high amount of computational power.<br />
<br />
Need to plan your analysis.<br />
<br />
'''NDSU Center for Computationally-Assisted Science and Technology (CCAST)''' provides a platform for these workloads connect via <code>ssh</code> uses loadable modules: <br />
<br />
* ''ex:'' <code>module load parallel</code> <br />
<br />
batch processing via PBS scripting<br />
</div><br />
<br />
== <span>Continued Learning</span> ==<br />
<div class="outline"><br />
=== General Programming/Computers Websites ===<br />
<br />
* StackOverflow.com - check before asking new questions <br />
* RosettaCode.org - data structures and algorithms in many languages <br />
* Linux.die.net/man/ - the Linux manual <br />
* grymoire.com/Unix/ - more *nix CLI tutorials<br />
<br />
=== Python ===<br />
<br />
* docs.python.org/3/ - the official python documentation <br />
* docs.python.org/3/tutorial - the official tutorial <br />
* diveintopython.net - guided tutorial online <br />
* pythontutor.com - visual debugger <br />
<br />
=== R ===<br />
<br />
* cran.r-project.org - CRAN <br />
* cran.r-project.org/manuals.html <br />
* rdrr.io - meta-manual lookup and many other tools for R <br />
* swirlstats.com - learn R, in R <br />
* statslearning.com - statistical machine learning coursework<br />
<br />
=== Recommended Reading ===<br />
<br />
* ''An Introduction to Statistical Machine Learning'' by Gareth James et al. <br />
* ''A Primer on Scientific Programming with Python'' by Hans Petter Langtangen <br />
* ''R for Data Science'' by Wickham and Grolmund</div></div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Main_Page&diff=139Main Page2022-10-21T20:29:11Z<p>Sysadmin: </p>
<hr />
<div><div><br />
=Welcome to the Rasulev Research Group Wiki!=<br />
==The Computational Polymer Science and Cheminformatics Group at North Dakota State University==<br />
<br />
This page hosts the tutorials and procedures for the group. Implementing MediaWiki, it provides a platform to organize knowledge and make it available to both lab collaborators and students. <br />
<div style="clear: both;"></div><br />
<br />
=== [[File:OOjs UI icon edit-ltr.svg|20px|middle|link=]]Topic Crash Courses ===<br />
<br />
* [[Statistical Machine Learning]]<br />
* [[Unix crash course|Unix Crash Course]]<div id="audiences" class="mainpage_row"><div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon chem.svg|20px|middle|link=]]</span><span>Lab Procedures and Software Tutorials</span></h3><br />
<div><br />
* [[ADF ReaxFF]]<br />
* [[Avogadro Procedure|Avogadro]]<br />
* [[BuildQsar Procedure|BuildQSAR]]<br />
* [[ChemDraw Procedure|ChemDraw]]<br />
* [[ChemSketchProcedure|ChemSketch]]<br />
* [[Dragon5 Procedure|Dragon5]]<br />
* [[Generating Descriptors Workflow]]<br />
* [[HyperChem]]<br />
* [[Jupyterlab anaconda|JupyterLab]]<br />
* [[OpenBabelGUI]]<br />
* [[QSARINS Procedure|QSARINS]]<br />
* [[VASP and VMD on CCAST|VASP]]<br />
* [[WEKA Steps for Loading Data|WEKA]]<br />
</div><br />
</div><br />
</div><div id="audiences" class="mainpage_row"><br />
<div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon advanced.svg|20px|middle|link=]]</span><span>Benchmarks and Findings</span></h3><br />
<div><br />
* [[Benchmarking NAMD workflows for GPU containers]]<br />
</div><br />
</div><br />
</div><br />
__NOEDITSECTION__<br />
__NOTOC__</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Statistical_Machine_Learning&diff=138Statistical Machine Learning2022-10-21T20:23:33Z<p>Sysadmin: created and edited page</p>
<hr />
<div>=== <span>Overview</span> ===<br />
<div class="outline"><br />
A crash course to enable gentle introduction into the machine learning techniques and its applications into data science. <br />
<br />
Topics Covered: <br />
<br />
* Basic Concepts <br />
* Data Models <br />
* Machine Learning <br />
* ML Techniques <br />
* SoftwareToolkit <br />
* Continued Learning<br />
<br />
== <span>Basic Concepts</span> ==<br />
<div class="outline"><br />
=== Definitions: ===<br />
<br />
* '''Statistical Machine Learning''' is a set of tools used to model and understand complex data sets <br />
* '''Data Science''' is a set of techniques in computing to support the analysis of data <br />
** Not very useful without some domain knowledge: it is important to ''know your data''. <br />
* Includes analytic techniques: <br />
** descriptive statistics <br />
** data visualization <br />
** statistical machine learning <br />
** neural networks<br />
** actor-environment models <br />
* Also includes computational techniques: <br />
** database administration <br />
** management of information systems <br />
** parallelization <math display="inline">\rightarrow</math> high performance computing<br />
</div><br />
<br />
=== Basic Concepts ===<br />
<div class="outline"><br />
==== Knowing your data ====<br />
Technical definition: <br />
<br />
Let <math display="inline">n</math> represent a number of distinct '''observations''', and let <math display="inline">p</math> represent a number of '''predictors''' Then, our observed data <math display="inline">\textbf{X}</math> is an <math display="inline">n\times p</math> matrix with row observation vectors <math display="inline">\vec{x}_{1..n}</math> and column predictor vectors <math display="inline">\vec{x}_{1..p}</math>. <br />
<br />
In addition, we will also have '''response''' variable(s) <math display="inline">\textbf{Y}</math>, which is a made up of some <math display="inline">n</math>-length vectors. <br />
<br />
So, our combined dataset consists of <math>[(\vec{x}_1,\vec{y}_1),(\vec{x}_2, \vec{y}_2),...,(\vec{x}_n,\vec{y}_n)]</math><br />
<br />
'''Our Mission''': determine relationships between <math display="inline">\textbf{X}</math> and <math display="inline">\textbf{Y}</math> which are mathematically sound, leading to better understandin<br />
<br />
Typically a table has columns as features, rows as entries<br />
<br />
'''Entries''' might be '''numeric''' or '''categorical'''. <br />
<br />
Data sources are either '''Structured''' or '''Unstructured''': <br />
<br />
* Unstructured data will require some transformation. <br />
<br />
Some data may also be '''time series''' taking a sampling of points over time, contributing to a 3-dimensional '''Data Cub'''<br />
<br />
Several techniques can be used to reduce complex data: <br />
<br />
* '''numeric representation''' mapping of categorical information into numbers. <br />
* '''scaling''' redefine a new range for a predictor vector. <br />
* '''normalization''' redefine a predictor by its mean and standard deviation, giving a normal distribution of values. <br />
* '''dimension reduction''' lose fine grain of data, but gain understandability. <br />
* '''feature extraction''' a data mining technique in which we can generate new predictors from known information<br />
<br />
==== Modeling ====<br />
What is a model?<br />
<br />
'''Very well-known model''': Gravity is a functional model between masses, distances, and force. <math display="block">F = G\frac{m_1m_2}{r^2} \rightarrow g = \frac{G M}{r^2} \rightarrow v(t) = v(0) - gt.</math> '''<math display="block">k = Ae^{\frac{E_a}{K_b T}}.</math>Statistics definition''': <br />
<br />
Let <math display="inline">X = (\vec{x}_1, \vec{x}_2, ..., \vec{x}_p)</math> each of length <math display="inline">n</math>, and <math display="inline">Y = (\vec{y})</math> of length <math display="inline">n</math>, then for <math display="inline">X</math> and <math display="inline">Y</math>, there exists a function with a '''systematic''' <math display="inline">f</math> and '''error term''' <math display="inline">\varepsilon</math>: <math display="block">Y = f(X) + \varepsilon</math>Why do we even estimate <math display="inline">f</math> at all? '''Prediction''' or '''Inference'''<br />
<br />
'''Predictive models''' create an estimator <math display="inline">\hat{f}</math> which we can use to estimate <math display="inline">Y</math> using a sample <math display="inline">X</math> from a larger population: <math display="block">\hat{ Y} = \hat{f}(X)</math> With error: <math display="inline">E(Y - \hat{Y})^2 = [f(X) - \hat{f}(X)]^2 + \varepsilon</math>. <br />
<br />
'''Inference models''' are primarily interested in how <math display="inline">Y</math> is affected by <math display="inline">X</math>: <br />
<br />
* What predictors associated with response? <br />
* What is the relationship of predictors to response? <br />
* What is the overall nature of relationship between <math display="inline">Y</math> and the predictors.<br />
<br />
==== Signal vs Noise ====<br />
Consider '''precision''' and '''accuracy'''. <br />
<br />
* Both contribute into data set <br />
<br />
'''noise:''' variation in data which detracts from constructing '''information''', as opposed to '''signal'''–data which is representative of a system under study and contains information. <br />
<br />
High signal to noise allows us to minimize '''reducible error''', caused by sampling technique. <br />
<br />
Different than '''irreducible error''', created by factors we are not measuring.<br />
<br />
==== Error and Fit ====<br />
In Modeling In the terms of modeling, precision of a model is referred to '''variance''' and the accuracy of a model its degree of '''bias'''. <br />
<br />
Generally, overly complex models generate high variance, and can '''over-fit''' to input data, making the model useless to new data.<br />
<br />
Generally, '''Mean Square Error''' or MSE, used to determine goodness-of-fit for model calibration: <math display="inline">MSE = \frac{1}{n}\sum^n_{i=1}(y_i - \hat{f}(x_i))^2</math>. <br />
<br />
Error rate used in classification: <math display="inline">\frac{1}{n} \sum^n_{i=1} I(y_i \ne \hat{y}_i)</math>. <br />
<br />
However for reporting, <math display="inline">R^2</math> statistic is more often used, because it gives a value between 0 and 1 useful to determine how much of variance in <math display="inline">Y</math> is explained by variance in <math display="inline">X</math>: <math display="inline">R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2} = \frac{RSS}{TSS}</math><br />
<br />
Example functions relating <math display="inline">X</math> and <math display="inline">Y</math>:<br />
<br />
* A linear function: <math display="inline">Y = \beta_0 + \beta_1 X + \varepsilon</math> <br />
* A polynomial: <math display="inline">Y = \beta_0 + \beta_1 X + \beta_2 X + ... + \varepsilon</math> <br />
* A natural function: <math display="inline">Y = e^{-\alpha_0 X^{\alpha_1}} + \varepsilon</math> <br />
* A logistical function: <math display="inline">p = \frac{e^{\beta_0 + \beta_1 X}}{1 + e^{\beta_0 + \beta_1 X}}</math> <br />
* A series of nested <code>if</code> statements <br />
* A series of differential equations: <math display="inline">x^\prime = x_n - \bar{x}_n: y^\prime = y_n - \beta_n ({x_n}^\prime).</math><br />
<br />
==== Why Machine Learning? ====<br />
Types of Questions: <br />
<br />
* '''Exact Solution is known''' normal coding problems, linear models, and classical statistics. <br />
* '''Exact Solution is unknown, but can be extracted with work''' work with systems experts and domain knowledge to create code. <br />
* '''Exact Solution is known, but not yet conveyable''' ML is useful. <br />
* '''Exact Solution not known by humans''' ML and/or Deep Learning needed<br />
<br />
''Example:'' consider a prediction of temperature: <br />
<br />
* '''Knowledge Based Models''' <br />
** Physics and atmospheric science based model. <br />
** Up to differential equations on chaotic systems. <br />
** As fine granularity of prediction increases, number of factors and density of data quickly becomes too much for most humans to consider <br />
* '''Data Driven Models''' <br />
** Use ML to provide '''iterative''' gain to reduce error <br />
*** Known data <math display="inline">\rightarrow</math> model creation <math display="inline">\rightarrow</math> point toward new factors.<br />
** Uses a split in training and testing data, or sum of error to move toward the correct answer.<br />
** Human researchers more free to find more data, improve prediction, develop theories<br />
<br />
== <span>Techniques</span> ==<br />
<br />
=== Machine Learning in General ===<br />
'''Supervised Learning:''' the model estimates, error verifies <br />
<br />
* If incorrect, needs user input for correction.<br />
* ''Example'': a computer vision system trained to find features in images via user annotated images. <br />
<br />
'''Unsupervised Learning:''' Clustering/Grouping of similar items<br />
<br />
* Need a similarity measure via feature vectors and ability to adjust weights <br />
* ''Example'': Taste prediction algorithms used in web advertising. <br />
<br />
'''Reinforced Learning''' Model estimates a sequence of guesses <br />
<br />
* Correct if and only if the entire sequence or a parameterized output scoring<br />
* Instant feedback but high compute cost <br />
* ''Example'': Game-play in actor-environment model<br />
<br />
=== Types of Problems and Output ===<br />
'''Numeric''' function maps <math display="inline">X</math> to <math display="inline">Y</math> and output is in <math display="inline">\mathbb{R}</math>. <br />
<br />
'''Categorization''' non-orderable sorting <br />
<br />
'''Clustering''' finding principle ways groups differ <br />
<br />
'''Anomaly Detection''' finding data points which are out of the ordinary <br />
<br />
'''Actor Models''' real-time decision making or detailed simulation<br />
<br />
=== Predictive Models ===<br />
Predictive ML models which are also Linear: <br />
<br />
* Utilize a split of training and test data: test-training or <math display="inline">k</math>-fold cross-validation <br />
* use a function mapping of one or more independent variables to the dependent variables, then re-evaluate to reduce error Includes techniques for mixed model reduction<br />
* Reduction of the number of predictors via '''Lasso''', '''Ridge''', and '''Elastic Net''' techniques<br />
<br />
=== Feature-vector based models ===<br />
Nested <code>if</code> statements try to find decision boundaries by distance between independent data and dependent outcome features have weighted probability most information by Bayesian inference<br />
<br />
* '''Decision Trees''' can be used to create decision models for linearly separable data effectively a neural network with one neuron<br />
* '''Random Forest''' utilizes a number of differently-tuned trees trees provide consensus voting-based approach for non-linearly separable data smaller tree depth typically prevents over-fit<br />
<br />
=== Clustering ===<br />
Groups data into cluster such that distance within clusters is small, and between differing groups is large <br />
<br />
Works with any well-defined &quot;distance&quot; function: Euclidean, Hamming, Inner Product, etc. <br />
<br />
'''k-Means Clustering:''' <br />
<br />
* choose <math display="inline">k</math> number of clusters randomly distribute <math display="inline">k</math> points, '''centroids''', into feature space <br />
* divide and classify data by distance to <math display="inline">k</math> centroids <br />
* move centroids based on center of groups repeats until convergence to some epsilon value<br />
* where points no longer move across iterations <br />
* '''Goodness of Fit''' for <math display="inline">N-M</math> possible values of <math display="inline">k</math>, an inflection in overall likelihood ratio given by probability function for set<br />
<br />
'''K-nearest Neighbors:''' <br />
<br />
Creates a probabilistic decision boundary within a feature space between <math display="inline">K</math> centroids<br />
<br />
Unsupervised system to find structures of data works on majority voting system<br />
<br />
'''Gradient Boosting:'''<br />
<br />
System attempts to find the direction and vector of change in a dimensional field, and follow these iteratively to find local extrema. <br />
<br />
Most use some '''Quasi-Newton Method''' for finding extrema for faster centroid convergence. <br />
<br />
'''Support Vector Machine:''' <br />
<br />
Also known as SVM, applies a classifier into high-dimensional data to split points into groups some use a '''kernel trick.'''<br />
<br />
Generates the inner product space of two arbitrary-dimensional numeric matrix spaces, showing the shape of data<br />
<br />
=== '''Genetic Algorithms''' ===<br />
Utilize some adversarial scoring method of initially randomized vectors:<br />
<br />
* ’survivors’ become the basis of new models similar iterative concept to gradient methods.<br />
* Does not have to understand topology of space requires creator to specify scoring for the machine .<br />
<br />
'''Convolutional Neural Networks:'''<br />
<br />
Utilize iterative scoring between training and testing, along with '''gradient descent''' on a number of layered, weighted vectors to extract features from a complex data set. <br />
<br />
'''Vision Systems''' <br />
<br />
Take high throughput data and simplify before work is done ''ex'': 1080p @ 60fps <math display="inline">\rightarrow</math> 240x360 @ 15fps, broken into component channels Gaussian blur filter kernel applied to high-res images Edge detection via double threshold ML or CNN used past this to determine actual features<br />
</div><br />
<br />
== Software Toolkit ==<br />
<div class="outline"><br />
=== Python ===<br />
General purpose programming language with many libraries Interpreted language: each line is run one at a time by a virtual machine.<br />
<br />
'''Dependency Structure''' <br />
<br />
''system'' vs ''user'' python <br />
<br />
virtualenv: <code>$ python3 -m venv /path/to/new/environment</code> <br />
<br />
pip libraries <code>$ pip list outdated format=freeze | grep -v | cut -d=" " -f1 | xargs -n1 pip install -U</code><br />
<br />
==== '''Anaconda''' ====<br />
separate virtualenv system specifically for data science:<br />
<br />
* '''Spyder''' IDE with visual output <br />
* '''JupyterLabs''' notes with data visualizations <br />
* '''Orange''' visual IDE for stats exploration<br />
* <code>pandas, NumPy, and SciPy</code> libraries for data serialization and numerical work in python <br />
* <code>matplotlib</code> for data visualization <br />
* <code>scikit-learn</code> main library for ML <br />
* <code>DASK</code> distributed abstraction layer with <code>pandas</code> grammar to easily distribute python tasks into 1-1000 compute nodes <br />
* <code>PyTorch, and TensorFlow</code> deep learning and CNN generation systems massive compute overhead to train models require data map reduction and or imputation to run well</div><div class="outline"><br />
=== R language ===<br />
Statistical programming language: interpreter invokes compiled C or FORTRAN.<br />
<br />
Also works within Jupyter notebook for instant visualization, if wanted.<br />
<br />
Open-source and extended by the Comprehensive R Archive Network (CRAN), which includes extensive documentation.<br />
<br />
* <code>rmarkdown</code> format a document from R with optional LaTeXbindings <br />
* <code>tidyverse</code> <br />
** <code>dplyr</code> grammar for mass data manipulation <br />
** <code>ggplot2</code> a library for creating graphs and visualizations <br />
* <code>doparallel</code> cost-free abstraction, pooling of CPU threads <br />
* <code>mlr</code> interface to a large number of classification and regression techniques <br />
* <code>shiny</code> provides ability to create web servers similar to NodeJS or Python Flaskl<br />
<br />
=== Intel MKL (Math Kernel Library) ===<br />
Improves performance for Fast Fourier Transforms, linear algebra operations, vector math, deep neural networks, and kernel solvers. <br />
<br />
Default math backend for NumPy, SciPy, and MATLAB <br />
<br />
Not hardware agnostic: chooses slowest solvers for non-Intel chips by default <br />
<br />
=== OpenBLAS and LAPACK ===<br />
'''LAPACK''' (Linear Algebra PACKage) provides APIs much like MKL <br />
<br />
'''OpenBLAS''' (Basic Linear Algebra Subprograms) extends LAPACK with optimizations for parallel computing <br />
<br />
Default for R and Biopython<br />
<br />
=== Message Passing Interface (MPI) ===<br />
Supported by all major compilers (Intel and OpenMP implementations) <br />
<br />
An API supporting shared-memory multiprocessing provides backend for many parallel computing systems, allowing for multi-threaded access <br />
<br />
''ex'': <code>mpirun -np $NUM_PROC /path/to/coolProgram &lt; $INPUT &gt; /path/to/output</code> <br />
<br />
=== GNU Parallel ===<br />
simple vectorization of loops over processors for non-multithreaded processes <br />
<br />
''ex'': <code>parallel -j $NUM_PROC /path/to/thescript.sh ::: </code><span><code>1..n</code></span><code> ::: </code><span><code>1..m</code></span><br />
<br />
=== CUDA/OpenCL ===<br />
Nvidia-specific '''CUDA''' and open-source '''OpenCL''' provide a hardware abstracting API for using GPU for compute tasks <br />
<br />
must-have for Pytorch or TensorFlow workloads <br />
<br />
Nomenclature Divergence <br />
<br />
* CUDA thread = OpenCL work item = CPU lane <br />
* CUDA multiprocessor = OpenCL compute unit = CPU<br />
<br />
=== High Performance Computers ===<br />
HPC or supercomputing clusters provide high throughput analysis.<br />
<br />
Amazingly high amount of computational power.<br />
<br />
Need to plan your analysis.<br />
<br />
'''NDSU Center for Computationally-Assisted Science and Technology (CCAST)''' provides a platform for these workloads connect via <code>ssh</code> uses loadable modules: <br />
<br />
* ''ex:'' <code>module load parallel</code> <br />
<br />
batch processing via PBS scripting<br />
</div><br />
<br />
== <span>Continued Learning</span> ==<br />
<div class="outline"><br />
=== General Programming/Computers Websites ===<br />
<br />
* StackOverflow.com - check before asking new questions <br />
* RosettaCode.org - data structures and algorithms in many languages <br />
* Linux.die.net/man/ - the Linux manual <br />
* grymoire.com/Unix/ - more *nix CLI tutorials<br />
<br />
=== Python ===<br />
<br />
* docs.python.org/3/ - the official python documentation <br />
* docs.python.org/3/tutorial - the official tutorial <br />
* diveintopython.net - guided tutorial online <br />
* pythontutor.com - visual debugger <br />
<br />
=== R ===<br />
<br />
* cran.r-project.org - CRAN <br />
* cran.r-project.org/manuals.html <br />
* rdrr.io - meta-manual lookup and many other tools for R <br />
* swirlstats.com - learn R, in R <br />
* statslearning.com - statistical machine learning coursework<br />
<br />
=== Recommended Reading ===<br />
<br />
* ''An Introduction to Statistical Machine Learning'' by Gareth James et al. <br />
* ''A Primer on Scientific Programming with Python'' by Hans Petter Langtangen <br />
* ''R for Data Science'' by Wickham and Grolmund</div></div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Benchmarking_NAMD_workflows_for_GPU_containers&diff=137Benchmarking NAMD workflows for GPU containers2022-10-21T18:22:52Z<p>Sysadmin: created and edited page</p>
<hr />
<div><span id="benchmarking-namd2-workloads-for-gpu-containers-on-ccast"></span><br />
== Benchmarking NAMD2 workloads for GPU containers on CCAST ==<br />
<br />
Stephen Szwiec <br />
<br />
Rasulev Computational Chemistry Research Group<br />
<br />
North Dakota State University<br />
<br />
26 July 2022<br />
-----<br />
==== ApoA1 information ====<br />
<br />
* simulates a bloodstream lipoprotein particle<br />
* '''physical stats'''<br />
** 92224 atoms<br />
** 70660 bonds<br />
** 74136 angles<br />
** 74130 diheadrals<br />
** 1402 impropers<br />
** 32992 hydrogen groups<br />
** 553785 total amu of mass<br />
* '''energy stats'''<br />
** 300K initial temp.<br />
** -14 e total charge<br />
* '''simulation stats'''<br />
** Consists of a startup process followed by 500 steps of simulation for benchmark time<br />
** GPU workload modified to use CUDA FFTW, CUDA integration<br />
** GPU workload modified to continue simulation for 10000 steps<br />
==== NAMD non-gpu run information ====<br />
<br />
* Charmrun used with one node (condo02)<br />
* machine topology: 2sockets x 64 cores x 1 PU = 128-way SMP possible<br />
* 20 processes, 20 cores, 1 physical node, and 16GB memory specified<br />
* NAMD 2.14 used<br />
==== NAMD gpu run information ====<br />
<br />
* NAMD used within Singularity container system<br />
** namd:3.0-alpha11 image used to generate singularity container<br />
* machine topology: 2sockets x 64 cores x 1 PU = 128-way SMP possible<br />
* 2 CPUs, 2 GPUs, 1 openMP thread, 1 physical node, and 16GB memory specified<br />
** binding each GPU to one Nvidia A10 with 22731MB memory<br />
* NAMD 3.0alpha11<br />
-----<br />
=== Benchmark findings ===<br />
==== Startup Wall Time ====<br />
<br />
'''CPU''' 3.4346 s '''GPU''' 0.0963 s<br />
==== Simulation Wall Time ====<br />
<br />
'''CPU''' 38.6991 s '''GPU''' 5.59571 s<br />
==== Wall Time Per Step ====<br />
<br />
'''CPU''' 0.0918308 s/step '''GPU''' 0.0110155 s/step<br />
==== Days Per Nanosecond Simulation ====<br />
<br />
'''CPU''' 0.999022 days/ns '''GPU''' 0.0660301 days/ns<br />
-----<br />
==== Additional information ====<br />
<br />
* Extending Apoa1 workload steps with CUDA lead to step 3500 being reached in roughly the same time step 500 was reached on 20 CPUs.<br />
* per core speedup with GPU acceleration is ~151.297 times<br />
** caveat: one CPU must supervise and await output from each GPU because of how NAMD was written<br />
* CUDA performed 10000steps and wrote output in 136.315155s total wall time<br />
* CPU performed 500steps and wrote output in 76.477325s total wall time</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Benchmarking_NAMD_workflows_for_GPU_containers&diff=136Benchmarking NAMD workflows for GPU containers2022-10-21T18:21:54Z<p>Sysadmin: Created page with "<span id="benchmarking-namd2-workloads-for-gpu-containers-on-ccast"></span> == Benchmarking NAMD2 workloads for GPU containers on CCAST == <span id="stephen-szwiec"></span> Stephen Szwiec Rasulev Computational Chemistry Research Group North Dakota State University 26 July 2022 ----- <span id="apoa1-information"></span> ==== ApoA1 information ==== * simulates a bloodstream lipoprotein particle * '''physical stats''' ** 92224 atoms ** 70660 bonds ** 74136 angles ** 741..."</p>
<hr />
<div><span id="benchmarking-namd2-workloads-for-gpu-containers-on-ccast"></span><br />
== Benchmarking NAMD2 workloads for GPU containers on CCAST ==<br />
<br />
<span id="stephen-szwiec"></span><br />
Stephen Szwiec <br />
Rasulev Computational Chemistry Research Group<br />
North Dakota State University<br />
26 July 2022<br />
-----<br />
<br />
<span id="apoa1-information"></span><br />
==== ApoA1 information ====<br />
<br />
* simulates a bloodstream lipoprotein particle<br />
* '''physical stats'''<br />
** 92224 atoms<br />
** 70660 bonds<br />
** 74136 angles<br />
** 74130 diheadrals<br />
** 1402 impropers<br />
** 32992 hydrogen groups<br />
** 553785 total amu of mass<br />
* '''energy stats'''<br />
** 300K initial temp.<br />
** -14 e total charge<br />
* '''simulation stats'''<br />
** Consists of a startup process followed by 500 steps of simulation for benchmark time<br />
** GPU workload modified to use CUDA FFTW, CUDA integration<br />
** GPU workload modified to continue simulation for 10000 steps<br />
<br />
<span id="namd-non-gpu-run-information"></span><br />
==== NAMD non-gpu run information ====<br />
<br />
* Charmrun used with one node (condo02)<br />
* machine topology: 2sockets x 64 cores x 1 PU = 128-way SMP possible<br />
* 20 processes, 20 cores, 1 physical node, and 16GB memory specified<br />
* NAMD 2.14 used<br />
<br />
<span id="namd-gpu-run-information"></span><br />
==== NAMD gpu run information ====<br />
<br />
* NAMD used within Singularity container system<br />
** namd:3.0-alpha11 image used to generate singularity container<br />
* machine topology: 2sockets x 64 cores x 1 PU = 128-way SMP possible<br />
* 2 CPUs, 2 GPUs, 1 openMP thread, 1 physical node, and 16GB memory specified<br />
** binding each GPU to one Nvidia A10 with 22731MB memory<br />
* NAMD 3.0alpha11<br />
<br />
<br />
-----<br />
<br />
<span id="benchmark-findings"></span><br />
=== Benchmark findings ===<br />
<br />
<span id="startup-wall-time"></span><br />
==== Startup Wall Time ====<br />
<br />
'''CPU''' 3.4346 s '''GPU''' 0.0963 s<br />
<br />
<span id="simulation-wall-time"></span><br />
==== Simulation Wall Time ====<br />
<br />
'''CPU''' 38.6991 s '''GPU''' 5.59571 s<br />
<br />
<span id="wall-time-per-step"></span><br />
==== Wall Time Per Step ====<br />
<br />
'''CPU''' 0.0918308 s/step '''GPU''' 0.0110155 s/step<br />
<br />
<span id="days-per-nanosecond-simulation"></span><br />
==== Days Per Nanosecond Simulation ====<br />
<br />
'''CPU''' 0.999022 days/ns '''GPU''' 0.0660301 days/ns<br />
<br />
<br />
-----<br />
<br />
<span id="additional-information"></span><br />
==== Additional information ====<br />
<br />
* Extending Apoa1 workload steps with CUDA lead to step 3500 being reached in roughly the same time step 500 was reached on 20 CPUs.<br />
* per core speedup with GPU acceleration is ~151.297 times<br />
** caveat: one CPU must supervise and await output from each GPU because of how NAMD was written<br />
* CUDA performed 10000steps and wrote output in 136.315155s total wall time<br />
* CPU performed 500steps and wrote output in 76.477325s total wall time<br />
<span id="benchmarking-namd2-workloads-for-gpu-containers-on-ccast"></span><br />
== Benchmarking NAMD2 workloads for GPU containers on CCAST ==<br />
<br />
<span id="stephen-szwiec"></span><br />
=== Stephen Szwiec ===<br />
<br />
<span id="rasulev-computational-chemistry-research-group"></span><br />
=== Rasulev Computational Chemistry Research Group ===<br />
<br />
<span id="north-dakota-state-university"></span><br />
=== North Dakota State University ===<br />
<br />
<span id="july-2022"></span><br />
=== 26 July 2022 ===<br />
<br />
<br />
-----<br />
<br />
<span id="apoa1-information"></span><br />
==== ApoA1 information ====<br />
<br />
* simulates a bloodstream lipoprotein particle<br />
* '''physical stats'''<br />
** 92224 atoms<br />
** 70660 bonds<br />
** 74136 angles<br />
** 74130 diheadrals<br />
** 1402 impropers<br />
** 32992 hydrogen groups<br />
** 553785 total amu of mass<br />
* '''energy stats'''<br />
** 300K initial temp.<br />
** -14 e total charge<br />
* '''simulation stats'''<br />
** Consists of a startup process followed by 500 steps of simulation for benchmark time<br />
** GPU workload modified to use CUDA FFTW, CUDA integration<br />
** GPU workload modified to continue simulation for 10000 steps<br />
<br />
<span id="namd-non-gpu-run-information"></span><br />
==== NAMD non-gpu run information ====<br />
<br />
* Charmrun used with one node (condo02)<br />
* machine topology: 2sockets x 64 cores x 1 PU = 128-way SMP possible<br />
* 20 processes, 20 cores, 1 physical node, and 16GB memory specified<br />
* NAMD 2.14 used<br />
<br />
<span id="namd-gpu-run-information"></span><br />
==== NAMD gpu run information ====<br />
<br />
* NAMD used within Singularity container system<br />
** namd:3.0-alpha11 image used to generate singularity container<br />
* machine topology: 2sockets x 64 cores x 1 PU = 128-way SMP possible<br />
* 2 CPUs, 2 GPUs, 1 openMP thread, 1 physical node, and 16GB memory specified<br />
** binding each GPU to one Nvidia A10 with 22731MB memory<br />
* NAMD 3.0alpha11<br />
<br />
<br />
-----<br />
<br />
<span id="benchmark-findings"></span><br />
=== Benchmark findings ===<br />
<br />
<span id="startup-wall-time"></span><br />
==== Startup Wall Time ====<br />
<br />
'''CPU''' 3.4346 s '''GPU''' 0.0963 s<br />
<br />
<span id="simulation-wall-time"></span><br />
==== Simulation Wall Time ====<br />
<br />
'''CPU''' 38.6991 s '''GPU''' 5.59571 s<br />
<br />
<span id="wall-time-per-step"></span><br />
==== Wall Time Per Step ====<br />
<br />
'''CPU''' 0.0918308 s/step '''GPU''' 0.0110155 s/step<br />
<br />
<span id="days-per-nanosecond-simulation"></span><br />
==== Days Per Nanosecond Simulation ====<br />
<br />
'''CPU''' 0.999022 days/ns '''GPU''' 0.0660301 days/ns<br />
<br />
<br />
-----<br />
<br />
<span id="additional-information"></span><br />
==== Additional information ====<br />
<br />
* Extending Apoa1 workload steps with CUDA lead to step 3500 being reached in roughly the same time step 500 was reached on 20 CPUs.<br />
* per core speedup with GPU acceleration is ~151.297 times<br />
** caveat: one CPU must supervise and await output from each GPU because of how NAMD was written<br />
* CUDA performed 10000steps and wrote output in 136.315155s total wall time<br />
* CPU performed 500steps and wrote output in 76.477325s total wall time</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Main_Page&diff=135Main Page2022-10-21T18:19:11Z<p>Sysadmin: </p>
<hr />
<div><div><br />
=Welcome to the Rasulev Research Group Wiki!=<br />
==The Computational Polymer Science and Cheminformatics Group at North Dakota State University==<br />
<br />
This page hosts the tutorials and procedures for the group. Implementing MediaWiki, it provides a platform to organize knowledge and make it available to both lab collaborators and students. <br />
<div style="clear: both;"></div><br />
<div id="audiences" class="mainpage_row"><br />
<div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon chem.svg|20px|middle|link=]]</span><span>Lab Procedures and Software Tutorials</span></h3><br />
<div><br />
* [[ADF ReaxFF]]<br />
* [[Avogadro Procedure|Avogadro]]<br />
* [[BuildQsar Procedure|BuildQSAR]]<br />
* [[ChemDraw Procedure|ChemDraw]]<br />
* [[ChemSketchProcedure|ChemSketch]]<br />
* [[Dragon5 Procedure|Dragon5]]<br />
* [[Generating Descriptors Workflow]]<br />
* [[HyperChem]]<br />
* [[Jupyterlab anaconda|JupyterLab]]<br />
* [[OpenBabelGUI]]<br />
* [[QSARINS Procedure|QSARINS]]<br />
* [[Unix crash course|Unix Crash Course]]<br />
* [[VASP and VMD on CCAST|VASP]]<br />
* [[WEKA Steps for Loading Data|WEKA]]<br />
</div><br />
</div><br />
</div><br />
<br />
<div style="clear: both;"></div><br />
<div id="audiences" class="mainpage_row"><br />
<div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon advanced.svg|20px|middle|link=]]</span><span>Benchmarks and Findings</span></h3><br />
<div><br />
* [[Benchmarking NAMD workflows for GPU containers]]<br />
</div><br />
</div><br />
</div><br />
__NOEDITSECTION__<br />
__NOTOC__</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Main_Page&diff=134Main Page2022-10-21T18:16:31Z<p>Sysadmin: </p>
<hr />
<div><div><br />
=Welcome to the Rasulev Research Group Wiki!=<br />
==The Computational Polymer Science and Cheminformatics Group at North Dakota State University==<br />
<br />
This page hosts the tutorials and procedures for the group. Implementing MediaWiki, it provides a platform to organize knowledge and make it available to both lab collaborators and students. <br />
<div style="clear: both;"></div><br />
<div id="audiences" class="mainpage_row"><br />
<div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon chem.svg|20px|middle|link=]]</span><span>Lab Procedures and Software Tutorials</span></h3><br />
<div><br />
* [[ADF ReaxFF]]<br />
* [[Avogadro Procedure|Avogadro]]<br />
* [[BuildQsar Procedure|BuildQSAR]]<br />
* [[ChemDraw Procedure|ChemDraw]]<br />
* [[ChemSketchProcedure|ChemSketch]]<br />
* [[Dragon5 Procedure|Dragon5]]<br />
* [[Generating Descriptors Workflow]]<br />
* [[HyperChem]]<br />
* [[Jupyterlab anaconda|JupyterLab]]<br />
* [[OpenBabelGUI]]<br />
* [[QSARINS Procedure|QSARINS]]<br />
* [[Unix crash course|Unix Crash Course]]<br />
* [[VASP and VMD on CCAST|VASP]]<br />
* [[WEKA Steps for Loading Data|WEKA]]<br />
</div><br />
</div><br />
</div><br />
<br />
__NOEDITSECTION__<br />
__NOTOC__</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Unix_crash_course&diff=133Unix crash course2022-10-21T18:14:31Z<p>Sysadmin: </p>
<hr />
<div>#Introduction to Unix<br />
<br />
<span id="filesystem"></span><br />
=== Filesystem ===<br />
<br />
<ol style="list-style-type: decimal;"><br />
<li>Directories<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>first and bottom of the directory of the file structure tree is called root '''/'''</li><br />
<li>'''/root''' in addition, there is a file directory, within the root directory called /root (“slash root”), which acts as root’s home directory</li><br />
<li>'''/home''' normal users’ directories</li><br />
<li>'''/boot''' contains the boot directory, including kernel, initial RAMdisk, and bootloader</li><br />
<li>'''/dev''' lists all devices of the computer</li><br />
<li>'''/etc''' configuration files for nearly all programs that need a state</li><br />
<li>'''/tmp''' temporary files for the system</li><br />
<li>'''/var''' variable-length files used by programs<br />
<ul><br />
<li>'''/var/spool''' old-school webmail location</li><br />
<li>'''/var/www''' location of http server files</li></ul><br />
</li><br />
<li>'''/usr''' files available to users, including binaries for daily user tasks in /usr/bin</li><br />
<li>'''/sbin''' binaries used only by root</li><br />
<li>'''/bin''' binaries for the system everyone needs to use</li></ol><br />
</li><br />
<li>Commands<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>'''cd''' change directories</li><br />
<li>'''~''' alias used to shortcut the current user directory</li><br />
<li>'''Ctrl+R''' search history of commands in <code>bash</code>; also available as output from <code>history</code></li><br />
<li>'''!!''' runs the previous command</li><br />
<li>'''ls''' list in the contents of current directory</li><br />
<li>'''.''': the current directory</li><br />
<li>'''..''' : go up one directory#User and Group Configuration</li></ol><br />
</li></ol><br />
<br />
<span id="the-shell-explained"></span><br />
=== The Shell explained ===<br />
<br />
# A shell is a command interpreter where the user interacts with the system via command line interface (CLI)<br />
# Different shells currently in-use A. '''Bourne-Again SHell''' - located at <code>/bin/bash</code> is the main shell and is used was a default on most Linux systems B. '''sh''' - The Bourne Shell, old school version does not require any libraries. Rarely seen as a user-interface shell; typically a symbolic link to another shell binary. Created by Stephen Bourne, first included in Unix V7 C. '''tcsh''' Tee Shell. Based on the earlier C-shell from Unix. Has some additional features built into it. D. '''csh''' C-shell, a BSD component not often seen on Linux. Created by Bill Joy at UC Berkeley. E. '''ksh''' Korn shell from Bell Labs in the 1980s. The idea was to incorporate features from the Bourne Shell along with C-programming syntax. F. '''zsh''' 1990s, incorporates additional ideas into ksh.<br />
# Built-in Commands A. <code>cd</code> changes directory - <code>cd ~</code> or <code>cd /home/user</code> are equivalent B. <code>ls -</code> lists files in a directory - default behavior just lists files - <code>-a</code> lists all files including hidden ones - <code>-l</code> long format - <code>-d</code> just directories - <code>-F</code> uses a special character to rep filetypes - <code>-R</code> recursive listing C. Single characters are substituted by <code>?</code>, wildcards by <code>*</code> D. Exiting the system - <code>exit</code> exits the current shell - <code>logout</code> only works for login shells E. Remote access - '''ssh''' or ''secure shell'' provides the basic command to create a shell across an encrypted pipe - <code>ssh username@remote.host</code> logs in as user ''username'' on a computer ''remote.host'' - several programs provide ssh access as a feature, such as PuTTY - '''scp''' or ''secure copy'' provides file transfer over ssh - <code>scp /path/to/localmachine/ username@remote.host:/path/to/destination</code> - <code>scp username@remote.host:/path/from/remotemachine /path/to/localdestination</code><br />
<br />
<span id="man-pages-and-getting-help"></span><br />
=== Man pages and getting help ===<br />
<br />
# displays the manual pages for any packages on the system<br />
# typing <code>[/]</code> will allow you to search through the page<br />
# allows you to move via <code>[spacebar]</code> or <code>[pgup/pgdn]</code><br />
# <code>man</code> is broken up into seperate chapters: <code>man 5</code> would show chapter 5<br />
## executable programs and shell<br />
## system calls<br />
## library calls<br />
## device calls<br />
## file formats<br />
## games<br />
## misc. (macro packages)<br />
## sys admin commands<br />
## kernel routines<br />
# <code>man -k *</code> : to find any pages matching <code>*</code><br />
# most programs also allow for <code>--help</code> or <code>-?</code> to be passed for a quick rundown on commands and usage<br />
<br />
<span id="unix-filesystem-permissions"></span><br />
=== Unix filesystem permissions ===<br />
<br />
<ol style="list-style-type: decimal;"><br />
<li>filesystem permissions saved as a series of 3 bit numbers</li><br />
<li>format: <code>d,rwx,rwx,rwx</code> for read-write-execute<br />
<ol style="list-style-type: lower-alpha;"><br />
<li><code>d</code> - denotes a Directory or file</li><br />
<li>first triad is for the User of the file</li><br />
<li>second triad is for Group members of the file</li><br />
<li>third triad is for Everyone else</li></ol><br />
</li><br />
<li>Read-write-execute for files<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>'''Read''' allows the file to be opened</li><br />
<li>'''Write''' allows for the file to be edited</li><br />
<li>'''Execute''' allows for the file to run as a program</li></ol><br />
</li><br />
<li>Read-write-execute for directories<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>'''Read''' allows for the directory to view filenames in the directory</li><br />
<li>'''Write''' allows files to be added or deleted from a dir</li><br />
<li>'''Execute''' allows the directory to be visited, “permission to traverse through”; must have this to execute programs within a directory</li></ol><br />
</li><br />
<li><code>chmod</code> changes file and directory permissions<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>two modes for chmod<br />
<ol style="list-style-type: decimal;"><br />
<li>'''symbolic mode''' <code>chmod {u,g,o}[+,-,=]{r,w,x}</code></li><br />
<li>absolute mode - chmod 777 /path/to/file` * 0 no permissions * 1 execute * 2 write * 4 read F. chown - change file owner and group</li><br />
<li>chown user /foo - changes foo’s owner to user</li><br />
<li>chown root:root /foo - changes foo’s owner to root, also changes foo’s group to root G. echo - displays any text you enter after the command</li><br />
<li>primarially used in shell scripts or ‘programs’ that utilize a combination of of built-in commands, variables, and executables to complete tasks</li><br />
<li>variables reached from echo by invoking the variable name as $NAME</li></ol><br />
</li></ol><br />
</li></ol><br />
<br />
<span id="redirection-symbols"></span><br />
=== Redirection symbols ===<br />
<br />
# prog &gt; - redirects std output to the file, truncating and rewriting<br />
# prog 2&gt; - redirect std error to the file, truncating and rewriting<br />
# &amp;&gt; - redirects both std output and error to file<br />
# <code>&gt;&gt;</code> - non-destructively appends to end of file rather than rewriting<br />
# <code>- &quot;backtick&quot; process commands in the middle of another command a. A tool similar to xargs b. text within backticks are treated as a separate command line who's results are substituted on this command line c. things inside of backticks are attempted to be used as a command d. example: rm</code> find / - user darren `<br />
# $() - is used like a backtick; more resistant to weird syntax<br />
<br />
<span id="piping-data-between-programs"></span><br />
=== Piping data between programs ===<br />
<br />
<ol style="list-style-type: decimal;"><br />
<li>programs on a UNIX system frequently use pipes to redirect each other’s output.</li><br />
<li>for instance you might use a text-filtering command on output from a file reading command.</li><br />
<li>allows small programs to operate on a flow of information in a row</li><br />
<li><p>- this is an unnamed pipe</p></li><br />
<li>mkfifo - allows for the creation of named pipes<br />
<ol style="list-style-type: lower-alpha;"><br />
<li><code>mkfifo fifo1</code></li><br />
<li><code>ls -l &gt; fifo1</code></li><br />
<li><code>cat &lt; fifo1</code></li></ol><br />
</li><br />
<li>tee - read from standard input and write to standard output and files<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>splits standard input so that is it is both displayed via standard output and also captured to one of more files</li><br />
<li>allows for logging and redirection while the process is also shown on screen</li><br />
<li>overwrites to a file as per ‘&gt;’ by default rather than like ‘&gt;&gt;’</li><br />
<li>example<br />
<ul><br />
<li>echo $PATH | tee file1 file2</li><br />
<li>file1, file2, and stdoutput all have same information</li></ul><br />
</li></ol><br />
</li><br />
<li>find - searches for files in directory hierarchy</li><br />
<li>xargs - builds a command from standard input, using command name and arguments to that command and then executes the command on each line of standard input<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>example: find / - user darren | xargs -d %rm</li></ol><br />
</li><br />
<li>file - returns the type of file and metadata information regarding files</li><br />
<li>join - combines two text files by matching the contents of specified fields<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>by default, join uses spaces as the limiters (unless -t CHAR is given)</li><br />
<li>example: join -1 3 -2 2<br />
<ul><br />
<li>this joins the third field of the first file and the second field of the second file</li></ul><br />
</li></ol><br />
</li><br />
<li>paste A. Paste is like cat rotated 90 degrees. When operating on multiple files, it will merge the lines of the files.</li><br />
<li>expand - converts tabs to spaces A. by default assumes you want 8 spaces per tab; -t changes B. unexpand - converts spaces to tabs</li><br />
<li>od - dump files into octal and other formats A. default usage is dumping binary file into octal binary format B. -x for 2byte hex output</li><br />
<li>sort - sorts lines of text files A. writes sorted concatenation of all files as output B. by default, uses first column C. Important options 1. -f : ignore case 2. -m : sort by three digit month abbreviation; ex, jan 3. -n : numeric sort 4. -v : sort by versio 5. -h : compare human readable numbers 6. -r : reverses the sort 7. -R : sorts randomly but groups like files 8. -d : dictionary-order 9. -k : sort via key, sorts by a field number, can be comma-seperated listq D. the opposite of this is ‘shuf’</li><br />
<li>split - splits a file into 2 or more files A. needs to have an output file prefix B. where do you want to split 1. -b : size in bytes 2. -l : lines C. example: split -l 2 listening1.txt numbers -&gt; output is: ‘numbersaa’ and ‘numbersab’ D. by default: splits every 10,000 lines</li><br />
<li>tr - trnaslate or delete characters A. changes individual characters coming in through standard input and outputs via standard output B. options 1. [:lower:] - all lower case chars 2. [:upper:] - all upper case chars 3. [::]</li><br />
<li>cut - extracts portions of lines of input and prints out as output A. used to remove items within lines, delimited by spaces or tabs B. options 1. -b : cuts the specified list by byte 2. -c : cuts the specified characters 3. -f : cuts specified list of fields from input 4. -d : specifies the delimiter for fields C. EXAMPLE: cut -f 7 -d : /etc/passwd</li><br />
<li>uniq - take an input list and report or omit repeated lines A. only if they are next to each other, so you have to sort first B. EXAMPLE: cut -f 7 -d : /etc/passwd | sort | uniq</li></ol><br />
<br />
<span id="tar-and-packaging"></span><br />
=== Tar and Packaging ===<br />
<br />
# tar A. the name ‘tar’ comes from its original purpose: Tape ARchive B. you can bundle files as .tar, ‘tarballs’, which contain a series of file entries and terminating bytes between files C. each entry has a file descriptor and the binary contents of the file D. file descriptor, or header, contains the name of a file, a checksum of that file, and permissions data E. does not compress data, but instead keeps it in one file F. tar options: -c: Creates a tar archive -v: Verbose output -x: eXtract from a tar archive -f: specify a Filename -A: Appends two archives together -M: create or extract a multivolume archive -j: specify bzip2 compression -J: specify xz compression -z: specify gzip compression –lzma: specify lzma compression -t: Test an archive<br />
<br />
<span id="more-unix-commands"></span><br />
=== More Unix Commands ===<br />
<br />
# fmt - format A. some files have crazy long lines of text B. reformats to a certain -width C. by default, truncates after 75 characters and cleans up paragraphs D. anything with 2+ blankline removed, and adds indention E. -s : prevents truncation of paragraphs<br />
# nl - line number A. nl does count whitespace by default B. identical to cat -b (but not cat -n)<br />
# pr - prepare a file for printing A. includes header, footer, and page breaks B. create colummns: pr -3 file.txt C. -l sets the length of lines D. -o choose the header text<br />
# regular expressions - sed and awk A. simliar to wildcard expansion: ex; ls ''.txt B. grep and sed are two examples of regEx programs C. unix system programs have two different forms of regular expressions: basic and extended D. types of expresssions: 1. exact text match: grep “blah” returns all lines with “blah” 2. bracketed expressions: b[aeiou]g - returns bag, beg, big, bog, bug 3. bracketed with range: a[2-4]z - returns a2z, a3z, a4z 4. matching single char: a.z (allows for anything between a and z) 5. matching begin/end of line: ^ beginning, $ end 6. any of these can be combined with the wildcard: '' ex: A.* Lincoln (with 0 or more occurances of a pattern.) ex: A[ae]* Lincoln (with 0 or more occurances of a-e after A. 7. matching at least one of: + symbol ex. A[ae]+ Lincoln - returns at least Aa or Ae Lincoln 8. multiple possibilities or: | needs to be set up with ’ ex. ‘Darren (F|Fredrick) Seifert’ 9. grep -E invokes regex functionality<br />
<br />
<span id="shell-scripting"></span><br />
=== Shell Scripting ===<br />
<br />
# The first line of a shell script tells which program to use to interpret the contents: #!/bin/bash or another shell A. the first two characters are (#!) are special code to tell the kernel this is a script - crunchbang, shebang, hashbang, poundbang B. the remaining lines beginning with # are considered comments <code># this is a comment</code><br />
# Classic Hello World<br />
<br />
<pre># hello.sh<br />
#!/bin/bash<br />
# Print out hello world<br />
echo 'Hello World!'</pre><br />
<ol start="3" style="list-style-type: decimal;"><br />
<li>When you want to run a shell script, it needs to be made executable before it is an executable file: ‘# chmod u+x ./file.sh’</li><br />
<li>Can be invoked with ./file.sh</li><br />
<li>Alternative routes: A. bash file.sh works whether or not #!/bin/bash or chmod +x has been given B. exec file.sh is used to run scripts from within other scripts and to stop the tty instance after it is done (used to run scripts within scripts)</li><br />
<li>Passing Parameters A. much like C++ programs, bash scripts can be passed parameters B. accessing these, the dollar sign is used, much like PHP or Perl, to call the number of the parameter 1. $0 - the name of the running script itself 2. $1 - the first parameter 3. and so on… 4. $? - returns the exit status of the last run command 5. $$ - returns the process ID (PID) of the current script 6. $USER - returns the username of the user running the script 7. $SECONDS - returns the runtime of the current script 8. $RANDOM - generates a random number (int 0 - 32000) 9. $LINENO - current line of script C. creation of new variables 1. variablename=value # no spaces ever 2. if you do want spaces, they must be escaped by quotes 3. double quotes recommended for variable substitution later 4. for instance varname=“value $1” allows for insertion of other variables into variables whereas varname=’value <math display="inline">1' will not work D. saving output into variables 1. output from programs can be saved into a variable 2. example: curdir=</math>(pwd) will run pwd and return the standard output as a variable 3. if you want to capture the standard error for a command, it can be saved as <code>error=$(&lt;/tmp/error)</code> 4. alternative use: curdir=<code>pwd</code> note the backtics</li></ol><br />
<br />
<pre># foo.sh<br />
#!/bin/bash<br />
# An example script<br />
cp $1 $2<br />
#verify it worked<br />
echo &quot;Details for $2&quot;<br />
ls -l $2</pre><br />
* handling user input and output can use ‘echo’ to create interactive scripts:<br />
<br />
<pre># plususer.sh<br />
#!/bin/bash<br />
echo -n 'Please enter user name for account:'<br />
read username # this saves the stdin input as username<br />
read -p 'Username' $username # -n no-newline<br />
echo $username<br />
read -sp 'Password:' password # -s stealth</pre><br />
* scripts can also handle files as standard input like other commands<br />
<br />
<pre># salesreport.sh<br />
#!/bin/bash<br />
#a sales report generated from standard input<br />
echo Data Summary:<br />
cat /dev/stdin | cut -d ' ' -F2,3 | sort</pre><br />
* variables can be made global via export<br />
<br />
<pre># script.sh<br />
#!/bin/bash<br />
var1=blah<br />
var2=foo<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
export var1<br />
exec ./script2<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre><br />
<pre>script2.sh<br />
#!/bin/bash<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
var1=flop<br />
var2=bleh<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Unix_crash_course&diff=132Unix crash course2022-10-21T18:03:24Z<p>Sysadmin: </p>
<hr />
<div>#Introduction to Unix<br />
<br />
<span id="syllabus"></span><br />
=== Syllabus ===<br />
<br />
<ol><br />
<li>Recommended course material:<br />
<ul><br />
<li>Unix and Linux System Administration 5th Edition - Evi Nemeth, et al.</li><br />
<li>CompTIA Linux+ Study Guide 4th Edition - Chritine Bresnahan and Richard Blum</li></ul><br />
</li><br />
<li>Objectives<br />
a. Shell scripting, input redirection, and C++/C code in a GNU/Linux environment<br />
b. understanding of Linux file system including permissions<br />
c. understanding of command and how to intermingle them to achieve more complex goals<br />
d. basic system architecture, installation process, and management of Linux/Unix</li><br />
<li>General notes<br />
a. Learning to use Bash scripting and <code>vi</code>/<code>neovim</code> essential to professional usage of Linux in real life<br />
b. Any commands discussed in class can be looked up via 'man'</li></ol><br />
<br />
##History<br />
###What is Unix?<br />
<br />
<ol><br />
<li>Bell System<br />
a. Dates back to late 1960s, Originally developed by Ken Thompson and Dennis Ritchie of Bell Labs<br />
b. Ritchie described the initial goal as to create a good environment in which to: - do programming and development - foster communal computing through remote access and close communication<br />
c. Unix originally written in assembly, but was later re-written in C in 1972<br />
d. first public releases in 1973, at Symposium on Operating Systems Principles; early source code sent by standard mail<br />
e. 1975, Version 6 of Unix was licensed to companies for the first time. However, commercial users were rare. Licensees only recieved source code from AT&amp;T.<br />
f. Also in 1975, Ken Thompson takes a sabbatical from Bell Labs as a professor at UC Berkeley. Creates BSD: The Berkeley System Distribution of Unix 4.2. - includes C-shell and vi editor for first time<br />
g. 1983: Bell broken into several companies in anti-trust suit against AT&amp;T.<br />
h. AT&amp;T new restrictive license almost kills commercial Unix. By 1991, BSD had grown to a complete rewrite of AT&amp;T Unix and comes with BSD license.<br />
i. BSD survives today as: FreeBSD, OpenBSD, NetBSD, etc. Additionally, Darwin kernel of Mac OSX continues as a form of Unix.</li><br />
<li>GNU Project into Linux<br />
a. Around 1983, Richard Stallman creates GNU project<br />
b. Stallman's GNU system goal was to create a 'free' system of software that had many of other capabilities of the commercial tools that many Unix also had<br />
c. By 1985, Stallman had founded the Free Software Foundation to raise funds to develop free versions of all the components of the Unix syste, under GNU GPL<br />
d. In 1991, Linus Torvalds began the development of a new free OS based on the MINIX operation system and SunOS. The 21 year old puts together the kernel.<br />
e. As the kernel for the GNU project was still very much in development, the two projects became a natural fit.<br />
f. To this day, Stallman insists that Linux is GNU/Linux due to the large reliance on GNU toolchain</li></ol><br />
<br />
#Introduction to Red Hat Enterprise Linux<br />
###Package Manager - YUM<br />
<br />
# stands for YellowDog Updater Modified<br />
# provides a set of tools for retrieving software from the internet using subscriptions to an update server<br />
# utilizes .rpm software to install packages for management<br />
# creates an Enterprise Linux environment<br />
#* cons: less software options overall<br />
#* pro: all available software is tested and works without additional configuration<br />
# <code>yum</code> has an additional, alternative wrapper program <code>dnf</code> to improve output and ease of use<br />
#* stands for DaNdified YUM<br />
#* commands in the format <code>sudo dnf [verb] [objects]</code><br />
#* <code>-y</code> flag gives YES as passed option for less attended installation<br />
# command commands<br />
#* <code>sudo dnf -y install [package_name]</code> install a particular package and dependencies<br />
#* <code>sudo dnf -y update</code> update all packages<br />
<br />
#Introduction to Ubuntu<br />
###Package Manager - APT<br />
<br />
<ol><br />
<li>stands for Advanced Package Manager</li><br />
<li>Ubuntu's set of tools to install packages and keep them up to date</li><br />
<li>utilizes .deb software packages for management</li><br />
<li>keep in mind this package management system is far from standard<br />
a. RPM is Linux Standard Base<br />
b. Many commerical Unixes have their own package managers<br />
c. when all else fails, a tarball (.tar) fille can be used to build from source</li><br />
<li>Commands<br />
a. <code>apt-get update</code> synchronizes the index of packages to your system<br />
b. <code>apt-get upgrade</code> downloads and installs new versions of all currently installed packages in place, based on the index<br />
c. <code>apt-get dist-upgrade</code> downloads and installs new versions, and moves the system up to the next version, based on the index<br />
d. <code>sudo</code> was originally &quot;superuserdo&quot;, now works to substitute - users and run programs as that user. /etc/sudoers controls this behavior.<br />
e. <code>sudo apt-get install foo</code> : installs package named '''foo'''<br />
f. <code>sudo apt-get remove bar</code> : removes a package named '''bar''', but not the config files<br />
g. <code>sudo apt-get purge bar</code> : removes a package named '''bar''', and removes config files 8. sudo apt-get source * : gets the source code for a package 9. sudo apt-get clean : removes downloaded packages (.deb files) used to install</li></ol><br />
<br />
<span id="filesystem"></span><br />
=== Filesystem ===<br />
<br />
<ol><br />
<li>Directories<br />
a. first and bottom of the directory of the file structure tree is called root '''/'''<br />
b. '''/root''' in addition, there is a file directory, within the root directory called /root (&quot;slash root&quot;), which acts as root's home directory<br />
c. '''/home''' normal users' directories<br />
d. '''/boot''' contains the boot directory, including kernel, initial RAMdisk, and bootloader<br />
e. '''/dev''' lists all devices of the computer<br />
f. '''/etc''' configuration files for nearly all programs that need a state<br />
g. '''/tmp''' temporary files for the system<br />
h. '''/var''' variable-length files used by programs - '''/var/spool''' old-school webmail location - '''/var/www''' location of http server files<br />
i. '''/usr''' files available to users, including binaries for daily user tasks in /usr/bin<br />
j. '''/sbin''' binaries used only by root<br />
k. '''/bin''' binaries for the system everyone needs to use</li><br />
<li>Commands<br />
a. '''cd''' change directories<br />
b. '''~''' alias used to shortcut the current user directory<br />
c. '''Ctrl+R''' search history of commands in <code>bash</code>; also available as output from <code>history</code><br />
d. '''!!''' runs the previous command<br />
e. '''ls''' list in the contents of current directory<br />
f. '''.''': the current directory<br />
g. '''..''' : go up one directory#User and Group Configuration</li></ol><br />
<br />
<span id="useraddusermod"></span><br />
=== useradd/usermod ===<br />
<br />
# <code>useradd</code> and <code>usermod</code> do much the same thing<br />
#* <code>-c</code> allows to add comments<br />
#* <code>-d</code> allows to change user home directory<br />
#* <code>-e</code> expiration date - date an account will expire (in YYYYMMDD format), kept in <code>/etc/shadow</code><br />
#* <code>-f</code> specify how many days the account can be inactive before password expires, kept in <code>/etc/shadow</code><br />
#* <code>-g</code> specify default group of the user<br />
#* <code>-G</code> append a list of additional groups for the user<br />
#* <code>-l</code>(usermod only) changes a user login ex: <code>usermod -l darren darseife</code><br />
#* <code>-u</code> uid<br />
#* <code>-s</code> specifies login shell<br />
#* <code>-p</code> specify pre-hashed password<br />
#* <code>-L</code> (usermod only) locks the user's password so it cannot be entered directly (best used with -e, as well)<br />
#* <code>-U</code> will unlock a user's password<br />
<br />
###passwd command revisited<br />
<br />
<ol><br />
<li><code>sudo passwd darseife</code> change the password of darseife</li><br />
<li><code>passwd</code> can also be used to lock and unlock accounts<br />
a. <code>passwd -l</code> will lock the account's password<br />
b. a locked password will always be input incorrectly<br />
c. does not necessarily lock user out of the account<br />
d. puts an <code>!</code> at the beginning of the password field in <code>/etc/shadow</code><br />
e. <code>crypt</code>, the algorithm used to implement ciphers, cannot generate an <code>!</code>so the password hash is disabled<br />
f. <code>passwd -u</code> would unlock the account's password<br />
g. <code>passwd -d</code> removes the password of the account<br />
h. <code>passwd -S</code> displays account information: - last password change - password expires - password inactive - account expires - minimum days between password change - maximum days before password change - warn period before changes</li></ol><br />
<br />
###chage<br />
<br />
# The <code>chage</code> command allows you to modify account settings related to password expiration and aging<br />
# <code>chage</code> can also be used to configure accounts to automatically expire if:<br />
#* the password hasn't been changed<br />
#* or a date has been passed<br />
# <code>chage -l</code> print expiration settings of an account<br />
# <code>chage -m DAYS username</code> to set the minimum time between password changes<br />
# <code>chage -M DAYS username</code> to set the maximum time between password changes<br />
# <code>chage -d YYYYMMDD username</code> update the last day YYYY-MM-DD on which to change the password<br />
# <code>chage -E YYYYMMDD username</code> set the expiration date as YYYY-MM-DD on which username's account expires<br />
#* setting the number to <code>-1</code> will make it never expire<br />
# <code>chage -I DAYS username</code> will lock the account password after number of days inactive<br />
# <code>chage -W DAYS username</code> number of days the user will be warned before their account is made inactive<br />
# All of these settings are stored in <code>/etc/shadow</code><br />
<br />
###userdel<br />
<br />
# <code>userdel username</code> deletes users<br />
# <code>userdel -r username</code> deletes the user, their mail spool, their home directory, and everything in it<br />
# <code>userdel -f username</code> deletes the user ''right now'' even if they are logged in<br />
<br />
###groupadd and groupmod<br />
<br />
# <code>groupadd</code>add a group to a system<br />
# <code>groupadd -r</code> used to create a system group, creating a GUID above 1000<br />
# <code>groupmod</code> change a group<br />
# <code>groupmod -n</code> change a group name<br />
<br />
###gpasswd - group password<br />
<br />
# <code>gpasswd groupname</code> sets password for a group; it is most common to not have a group password<br />
# <code>gpasswd -R</code> restricts access to users logging in with password via newgrp<br />
<br />
###getent - get entities<br />
<br />
<ol><br />
<li>can be used to view individual records in <code>/etc/passwd</code>, and other files</li><br />
<li>useful for getting exact matches for items</li><br />
<li><code>getent password username</code> - get passwd information for user</li><br />
<li><code>getent shadow username</code> - get shadow information for user</li><br />
<li><code>getent group groupname</code> - get group information for user#The *nix CLI<br />
###The Shell explained</li><br />
<li>A shell is a command interpreter where the user interacts with the system via command line interface (CLI)</li><br />
<li>Different shells currently in-use A. '''Bourne-Again SHell''' - located at <code>/bin/bash</code> is the main shell and is used was a default on most Linux systems B. '''sh''' - The Bourne Shell, old school version does not require any libraries. Rarely seen as a user-interface shell; typically a symbolic link to another shell binary. Created by Stephen Bourne, first included in Unix V7 C. '''tcsh''' Tee Shell. Based on the earlier C-shell from Unix. Has some additional features built into it. D. '''csh''' C-shell, a BSD component not often seen on Linux. Created by Bill Joy at UC Berkeley. E. '''ksh''' Korn shell from Bell Labs in the 1980s. The idea was to incorporate features from the Bourne Shell along with C-programming syntax. F. '''zsh''' 1990s, incorporates additional ideas into ksh.</li><br />
<li>Built-in Commands A. <code>cd</code> changes directory - <code>cd ~</code> or <code>cd /home/user</code> are equivalent B. <code>ls -</code> lists files in a directory - default behavior just lists files - <code>-a</code> lists all files including hidden ones - <code>-l</code> long format - <code>-d</code> just directories - <code>-F</code> uses a special character to rep filetypes - <code>-R</code> recursive listing C. Single characters are substituted by <code>?</code>, wildcards by <code>*</code> D. Exiting the system - <code>exit</code> exits the current shell - <code>logout</code> only works for login shells E. Shutting down the system - <code>shutdown -h now</code> halts the system now - <code>shutdown -r now</code> restarts the system now - <code>shutdown -h +5 &quot; &quot;</code> broadcasts the message in quotes and tells people it will be down in 5minutes - <code>halt</code> on most systems, will drop the system into single user mode - <code>poweroff</code> &quot;on fire&quot; graceless exit. Sends HW signal to kill power - <code>reboot</code> essentially 'shutdown -r' - <code>reboot -f</code> graceless restart exit, Sends HW signal to reboot F. <code>init</code> relies on a series of programs or scripts to change run-state of machine and to start/stop linux services - <code>init 0</code> - shuts the machine down - <code>init 6</code> - reboots the machine - <code>init S</code> - initializes system boot</li></ol><br />
<br />
###Man pages and getting help<br />
<br />
# displays the manual pages for any packages on the system<br />
# typing <code>[/]</code> will allow you to search through the page<br />
# allows you to move via <code>[spacebar]</code> or <code>[pgup/pgdn]</code><br />
# <code>man</code> is broken up into seperate chapters: <code>man 5</code> would show chapter 5<br />
## executable programs and shell<br />
## system calls<br />
## library calls<br />
## device calls<br />
## file formats<br />
## games<br />
## misc. (macro packages)<br />
## sys admin commands<br />
## kernel routines<br />
# <code>man -k *</code> : to find any pages matching <code>*</code><br />
# most programs also allow for <code>--help</code> or <code>-?</code> to be passed for a quick rundown on commands and usage<br />
<br />
###Unix filesystem permissions<br />
<br />
<ol><br />
<li>filesystem permissions saved as a series of 3 bit numbers</li><br />
<li>format: <code>d,rwx,rwx,rwx</code> for read-write-execute<br />
a. <code>d</code> - denotes a Directory or file<br />
b. first triad is for the User of the file<br />
c. second triad is for Group members of the file<br />
d. third triad is for Everyone else</li><br />
<li>Read-write-execute for files<br />
a. '''Read''' allows the file to be opened<br />
b. '''Write''' allows for the file to be edited<br />
c. '''Execute''' allows for the file to run as a program</li><br />
<li>Read-write-execute for directories<br />
a. '''Read''' allows for the directory to view filenames in the directory<br />
b. '''Write''' allows files to be added or deleted from a dir<br />
c. '''Execute''' allows the directory to be visited, &quot;permission to traverse through&quot;; must have this to execute programs within a directory</li><br />
<li><code>chmod</code> changes file and directory permissions<br />
a. two modes for chmod 1. '''symbolic mode''' <code>chmod {u,g,o}[+,-,=]{r,w,x}</code> 2. absolute mode - chmod 777 /path/to/file` * 0 no permissions * 1 execute * 2 write * 4 read F. chown - change file owner and group 1. chown user /foo - changes foo's owner to user 2. chown root:root /foo - changes foo's owner to root, also changes foo's group to root G. echo - displays any text you enter after the command 1. primarially used in shell scripts or 'programs' that utilize a combination of of built-in commands, variables, and executables to complete tasks 2. variables reached from echo by invoking the variable name as $NAME<br />
#Input Redirection and Utils</li></ol><br />
<br />
###Redirection symbols<br />
<br />
# prog &gt; - redirects std output to the file, truncating and rewriting<br />
# prog 2&gt; - redirect std error to the file, truncating and rewriting<br />
# &amp;&gt; - redirects both std output and error to file<br />
# <code>&gt;&gt;</code> - non-destructively appends to end of file rather than rewriting<br />
# <code>- &quot;backtick&quot; process commands in the middle of another command a. A tool similar to xargs b. text within backticks are treated as a separate command line who's results are substituted on this command line c. things inside of backticks are attempted to be used as a command d. example: rm</code> find / - user darren `<br />
# $() - is used like a backtick; more resistant to weird syntax<br />
<br />
###Piping data between programs<br />
<br />
<ol><br />
<li>programs on a UNIX system frequently use pipes to redirect each other's output.</li><br />
<li>for instance you might use a text-filtering command on output from a file reading command.</li><br />
<li>allows small programs to operate on a flow of information in a row</li><br />
<li>| - this is an unnamed pipe</li><br />
<li>mkfifo - allows for the creation of named pipes<br />
a. <code>mkfifo fifo1</code><br />
b. <code>ls -l &gt; fifo1</code><br />
c. <code>cat &lt; fifo1</code></li><br />
<li>tee - read from standard input and write to standard output and files<br />
a. splits standard input so that is it is both displayed via standard output and also captured to one of more files<br />
b. allows for logging and redirection while the process is also shown on screen<br />
c. overwrites to a file as per '&gt;' by default rather than like '&gt;&gt;'<br />
d. example - echo $PATH | tee file1 file2 - file1, file2, and stdoutput all have same information</li><br />
<li>find - searches for files in directory hierarchy</li><br />
<li>xargs - builds a command from standard input, using command name and arguments to that command and then executes the command on each line of standard input<br />
a. example: find / - user darren | xargs -d %\n rm</li><br />
<li>file - returns the type of file and metadata information regarding files</li><br />
<li>join - combines two text files by matching the contents of specified fields<br />
a. by default, join uses spaces as the limiters (unless -t CHAR is given)<br />
b. example: join -1 3 -2 2 - this joins the third field of the first file and the second field of the second file</li><br />
<li>paste A. Paste is like cat rotated 90 degrees. When operating on multiple files, it will merge the lines of the files.</li><br />
<li>expand - converts tabs to spaces A. by default assumes you want 8 spaces per tab; -t changes B. unexpand - converts spaces to tabs</li><br />
<li>od - dump files into octal and other formats A. default usage is dumping binary file into octal binary format B. -x for 2byte hex output</li><br />
<li>sort - sorts lines of text files A. writes sorted concatenation of all files as output B. by default, uses first column C. Important options 1. -f : ignore case 2. -m : sort by three digit month abbreviation; ex, jan 3. -n : numeric sort 4. -v : sort by versio 5. -h : compare human readable numbers 6. -r : reverses the sort 7. -R : sorts randomly but groups like files 8. -d : dictionary-order 9. -k : sort via key, sorts by a field number, can be comma-seperated listq D. the opposite of this is 'shuf'</li><br />
<li>split - splits a file into 2 or more files A. needs to have an output file prefix B. where do you want to split 1. -b : size in bytes 2. -l : lines C. example: split -l 2 listening1.txt numbers -&gt; output is: 'numbersaa' and 'numbersab' D. by default: splits every 10,000 lines</li><br />
<li>tr - trnaslate or delete characters A. changes individual characters coming in through standard input and outputs via standard output B. options 1. [:lower:] - all lower case chars 2. [:upper:] - all upper case chars 3. [::]</li><br />
<li>cut - extracts portions of lines of input and prints out as output A. used to remove items within lines, delimited by spaces or tabs B. options 1. -b : cuts the specified list by byte 2. -c : cuts the specified characters 3. -f : cuts specified list of fields from input 4. -d : specifies the delimiter for fields C. EXAMPLE: cut -f 7 -d : /etc/passwd</li><br />
<li>uniq - take an input list and report or omit repeated lines A. only if they are next to each other, so you have to sort first B. EXAMPLE: cut -f 7 -d : /etc/passwd | sort | uniq</li></ol><br />
<br />
<span id="c-programs-and-makefiles"></span><br />
=== C++ programs and Makefiles ===<br />
<br />
# Basic C++ program with input and output<br />
<br />
<pre>int main (int args, char * argv[]){<br />
for(int x = 0; x &lt; argv[],x++){<br />
cout &lt;&lt; arv[x] &lt;&lt; endl;<br />
}<br />
return 0;<br />
}<br />
// outputs every character given to it</pre><br />
# Let's write a less good version of grep called greb:<br />
<br />
<pre>#include &lt;iostream&gt;<br />
#include &lt;fstream&gt;<br />
using namespace std;<br />
<br />
int main(int argc, char * argv[]) {<br />
string curLine;<br />
if (argc == 2){<br />
while(getline(cin,curLine)) {<br />
if (curLine.find(argv[1]) != string::npos) {<br />
cout &lt;&lt; curLine &lt;&lt; endl;<br />
}<br />
}<br />
}<br />
<br />
else if (argc == 3){<br />
ifstream inFile(argv[2], ifstream::in);<br />
<br />
while(getline(inFile,curLine)) {<br />
if (curLine.find(argv[1]) != string::npos) {<br />
cout &lt;&lt; curLine &lt;&lt; endl;<br />
}<br />
}<br />
}<br />
else {<br />
cerr &lt;&lt; &quot;Error: unknown parameters.&quot; &lt;&lt; endl;<br />
return 1;<br />
}<br />
return 0;<br />
}<br />
</pre><br />
# Make A. Allows for automatic configuration and creation of usable binararies from OOP files, and can be used to install and uninstall programs B. Allows for the creation of archives of source code that build as packages into useful code C. typically named 'makefile'<br />
<br />
<pre># set a variables for the compiler<br />
CC = g++<br />
LC_MESSAGES=C<br />
CFLAGS=-g -Wall<br />
<br />
# target name variable<br />
TARGET = greb<br />
<br />
# file extension<br />
FILEXT = cpp<br />
<br />
# the build target executable =&gt; install path of the executable<br />
PREFIX = /usr/local<br />
DESTDIR = /home/bernd/staging/<br />
<br />
# basic makefile format<br />
# name of instruction: dependencies<br />
# &lt;tab&gt; command to execute<br />
# the FIRST one you list is the default<br />
$(TARGET): $(TARGET).$(FILEXT)<br />
$(CC) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)<br />
<br />
.PHONY: all<br />
all: $(TARGET).$(FILEXT)<br />
$(CC) $(MAKEOPTS) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)<br />
<br />
.PHONY: clean<br />
clean:<br />
rm $(TARGET)<br />
<br />
.PHONY: install<br />
install: $(TARGET)<br />
mkdir -p $(DESTDIR)$(PREFIX)/bin<br />
cp $&lt; $(DESTDIR)$(PREFIX)/bin/$&lt;</pre><br />
<span id="tar-and-packaging"></span><br />
=== Tar and Packaging ===<br />
<br />
# tar A. the name 'tar' comes from its original purpose: Tape ARchive B. you can bundle files as .tar, 'tarballs', which contain a series of file entries and terminating bytes between files C. each entry has a file descriptor and the binary contents of the file D. file descriptor, or header, contains the name of a file, a checksum of that file, and permissions data E. does not compress data, but instead keeps it in one file F. tar options: -c: Creates a tar archive -v: Verbose output -x: eXtract from a tar archive -f: specify a Filename -A: Appends two archives together -M: create or extract a multivolume archive -j: specify bzip2 compression -J: specify xz compression -z: specify gzip compression --lzma: specify lzma compression -t: Test an archive<br />
<br />
<span id="more-unix-commands"></span><br />
=== More Unix Commands ===<br />
<br />
# fmt - format A. some files have crazy long lines of text B. reformats to a certain -width C. by default, truncates after 75 characters and cleans up paragraphs D. anything with 2+ blankline removed, and adds indention E. -s : prevents truncation of paragraphs<br />
# nl - line number A. nl does count whitespace by default B. identical to cat -b (but not cat -n)<br />
# pr - prepare a file for printing A. includes header, footer, and page breaks B. create colummns: pr -3 file.txt C. -l sets the length of lines D. -o choose the header text<br />
# regular expressions - sed and awk A. simliar to wildcard expansion: ex; ls ''.txt B. grep and sed are two examples of regEx programs C. unix system programs have two different forms of regular expressions: basic and extended D. types of expresssions: 1. exact text match: grep &quot;blah&quot; returns all lines with &quot;blah&quot; 2. bracketed expressions: b[aeiou]g - returns bag, beg, big, bog, bug 3. bracketed with range: a[2-4]z - returns a2z, a3z, a4z 4. matching single char: a.z (allows for anything between a and z) 5. matching begin/end of line: ^ beginning, $ end 6. any of these can be combined with the wildcard: '' ex: A.* Lincoln (with 0 or more occurances of a pattern.) ex: A[ae]* Lincoln (with 0 or more occurances of a-e after A. 7. matching at least one of: + symbol ex. A[ae]+ Lincoln - returns at least Aa or Ae Lincoln 8. multiple possibilities or: | needs to be set up with ' ex. 'Darren (F|Fredrick) Seifert' 9. grep -E invokes regex functionality<br />
<br />
<span id="shell-scripting"></span><br />
=== Shell Scripting ===<br />
<br />
# The first line of a shell script tells which program to use to interpret the contents: #!/bin/bash or another shell A. the first two characters are (#!) are special code to tell the kernel this is a script - crunchbang, shebang, hashbang, poundbang B. the remaining lines beginning with # are considered comments <code># this is a comment</code><br />
# Classic Hello World<br />
<br />
<pre># hello.sh<br />
#!/bin/bash<br />
# Print out hello world<br />
echo 'Hello World!'</pre><br />
# When you want to run a shell script, it needs to be made executable before it is an executable file: '# chmod u+x ./file.sh'<br />
# Can be invoked with ./file.sh<br />
# Alternative routes: A. bash file.sh works whether or not #!/bin/bash or chmod +x has been given B. exec file.sh is used to run scripts from within other scripts and to stop the tty instance after it is done (used to run scripts within scripts)<br />
# Passing Parameters A. much like C++ programs, bash scripts can be passed parameters B. accessing these, the dollar sign is used, much like PHP or Perl, to call the number of the parameter 1. $0 - the name of the running script itself 2. $1 - the first parameter 3. and so on... 4. $? - returns the exit status of the last run command 5. $$ - returns the process ID (PID) of the current script 6. $USER - returns the username of the user running the script 7. $SECONDS - returns the runtime of the current script 8. $RANDOM - generates a random number (int 0 - 32000) 9. $LINENO - current line of script C. creation of new variables 1. variablename=value # no spaces ever 2. if you do want spaces, they must be escaped by quotes 3. double quotes recommended for variable substitution later 4. for instance varname=&quot;value $1&quot; allows for insertion of other variables into variables whereas varname='value $1' will not work D. saving output into variables 1. output from programs can be saved into a variable 2. example: curdir=$(pwd) will run pwd and return the standard output as a variable 3. if you want to capture the standard error for a command, it can be saved as <code>error=$(&lt;/tmp/error)</code> 4. alternative use: curdir=<code>pwd</code> note the backtics<br />
<br />
<pre># foo.sh<br />
#!/bin/bash<br />
# An example script<br />
cp $1 $2<br />
#verify it worked<br />
echo &quot;Details for $2&quot;<br />
ls -l $2</pre><br />
* handling user input and output can use 'echo' to create interactive scripts:<br />
<br />
<pre># plususer.sh<br />
#!/bin/bash<br />
echo -n 'Please enter user name for account:'<br />
read username # this saves the stdin input as username<br />
read -p 'Username' $username # -n no-newline<br />
echo $username<br />
read -sp 'Password:' password # -s stealth</pre><br />
* scripts can also handle files as standard input like other commands<br />
<br />
<pre># salesreport.sh<br />
#!/bin/bash<br />
#a sales report generated from standard input<br />
echo Data Summary:<br />
cat /dev/stdin | cut -d ' ' -F2,3 | sort</pre><br />
* variables can be made global via export<br />
<br />
<pre># script.sh<br />
#!/bin/bash<br />
var1=blah<br />
var2=foo<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
export var1<br />
exec ./script2<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre><br />
<pre>script2.sh<br />
#!/bin/bash<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
var1=flop<br />
var2=bleh<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre><br />
<span id="sed---stream-editor"></span><br />
=== Sed - Stream EDitor ===<br />
<br />
<ol><br />
<li>the sed command uses regular expressions to modify stdin and sends the changes to stdout</li><br />
<li>usage : <code>sed [options] script.txt [inputfile]</code></li><br />
<li>some of the commands that can be used in the script portion<br />
<ul><br />
<li><code>=</code> - display the current line number</li><br />
<li><code>a\text</code> - append text to file</li><br />
<li><code>i\text</code> - insert text to file</li><br />
<li><code>r file</code> - append text from filename into the file</li><br />
<li><code>c\text</code> - replace the selected range with the text</li><br />
<li><code>s/2012/2013/</code> - replace the first occurrence of 2012 on first occurrence on each line</li><br />
<li><code>s/2012/2013/g</code> - replace '' '' every occurrence on each line</li></ul><br />
</li><br />
<li>examples<br />
<ul><br />
<li><p><code>sed 's/Nick\|nick/Gary/g' &lt; salaryprojections.txt</code></p></li><br />
<li><p>^ this would replace any 'Nick' or 'nick' with 'Gary' at all points in the file</p></li><br />
<li><p>standard replace format: <code>sed 's/^/ /' file.txt &gt; file.new.txt</code></p><br />
<span id="examining-processes"></span><br />
=== Examining Processes ===<br />
</li></ul><br />
</li><br />
<li><code>ps</code> command<br />
<ul><br />
<li>The internal Unix command for process control is <code>ps</code></li><br />
<li>By default, <code>ps</code> reports back what is happening inside of the current terminal/tty;</li><br />
<li>shows <code>PID</code>, <code>TTY</code>, <code>TIME</code>, and <code>CMD</code><br />
<ul><br />
<li><code>PID</code> is the Process ID number for a running program</li><br />
<li><code>TTY</code> is the Teletypewriter device which the process runs from; reports stdout and stderr to this by default</li><br />
<li>TIME is the cumulative processor time used by the program on Linux</li></ul><br />
</li><br />
<li><code>ps</code> supports traditional Unix options and GNU flags, which can be somewhat confusing</li><br />
<li>Options for ps<br />
<ul><br />
<li>Display all processes running <code>ps -A/-e</code> both cause all processes on the system to be listed and output is identical</li><br />
<li>Display processes belonging to a specific user: <code>ps -u user</code></li><br />
<li>Display extra fields as a “full format” <code>ps -f</code> can be combined with other options<br />
<ul><br />
<li><code>UID</code> User ID shows the username of the user running the command</li><br />
<li><code>PPID</code> parent process ID, the process ID of the process that spawned the process</li><br />
<li><code>C</code> processor utilization as % of usage over the lifetime of the process</li><br />
<li><code>STIME</code> start time of the process as per system time (only in 24h clock format if current day; otherwise uses MMDD format)</li></ul><br />
</li><br />
<li>Display long format <code>ps -l</code><br />
<ul><br />
<li><code>F</code> Flags field<br />
<ul><br />
<li><code>0</code> = no special flags for this process</li><br />
<li><code>1</code> = forked process, but did not call exec</li><br />
<li><code>4</code> = executable was executed with superuser privileges</li></ul><br />
</li><br />
<li><code>S</code> State codes<br />
<ul><br />
<li><code>D</code> Uninterruptible sleep (usually IO)</li><br />
<li><code>R</code> Running or runnable (on run queue)</li><br />
<li><code>S</code> interruptible Sleep (waiting for an event to complete)</li><br />
<li><code>T</code> Stopped, either by a job control signal or because it is being traced.</li><br />
<li><code>X</code> dead (should never be seen)</li><br />
<li><code>Z</code> Defunct (“zombie”) process, terminated but not reaped by its parent.</li></ul><br />
</li><br />
<li><code>UID</code> uses system number rather than human readable name</li><br />
<li><code>PRI</code> Priority is used by the kernel, rather than in userspace, to determine the next in queue for processing</li><br />
<li><code>NI</code> Niceness of a process describes how accommodating a process is to sharing time on the system.<br />
<ul><br />
<li>This number ranges from +19 to -20, with 0 as the default.</li><br />
<li>In the scheduling system, niceness is used to determine how much processing to give to each process when it asks for it.</li></ul><br />
</li><br />
<li><code>ADDR</code> the memory address of the process</li><br />
<li><code>SZ</code> the size of the image in pages<br />
<ul><br />
<li>how much virtual memory is in use by the process.</li><br />
<li>Includes text, data, and stack space</li><br />
<li>page size is a kernel variable and can be read by <code>getconf PAGE_SIZE</code></li></ul><br />
</li><br />
<li><code>WCHAN</code> address of where a function is sleeping (only kernel functions)</li><br />
<li><code>RSS</code> amount of physical memory in use</li></ul><br />
</li></ul><br />
</li><br />
<li>User-oriented format <code>ps -u</code> displays information in a more readable manner<br />
<ul><br />
<li><code>%CPU</code> percent of processor time used by process divided by real time</li><br />
<li><code>%MEM</code> actual percentage of non-reserved physical RAM</li><br />
<li><code>VSIZE</code> virtual memory size used</li></ul><br />
</li><br />
<li>Virtual memory format <code>ps -v</code> displays information including number of:<br />
<ul><br />
<li><code>MAJFL</code> major page faults of the process, indicating times it has used swap space</li><br />
<li><code>TRS</code> physical memory devoted to stack</li><br />
<li><code>DRS</code> physical memory devoted to non-executable code</li></ul><br />
</li><br />
<li>Recieve a process hierarchy <code>ps -AH</code> see parent-child relationships visually</li><br />
<li>Choose your own adventure mode <code>ps -o uid,pid,ppid,command</code><br />
<ul><br />
<li>this command used to determine certain columns</li></ul><br />
</li><br />
<li>Piping ps can be piped as per normal to other programs:<br />
<ul><br />
<li>Ex; <code>ps -u darseife \| grep bash</code></li><br />
<li><code>pgrep</code> is a common utility that behaves similarly to the previous command</li></ul><br />
</li></ul><br />
</li><br />
<li><code>top</code><br />
<ul><br />
<li><code>top</code> is used to view processes as they execute in real time<br />
<ul><br />
<li><code>-d</code> used to choose update frequency in seconds</li><br />
<li><code>-p</code> pid to monitor specifically</li><br />
<li><code>-n</code> run for N iterations</li><br />
<li><code>-r</code> command to renice a process within top</li><br />
<li><code>-s</code> to change the update rate within top</li></ul><br />
</li><br />
<li><code>htop</code> can also be used as a next generation to top</li></ul><br />
</li><br />
<li><code>nice</code> and <code>renice</code><br />
<ul><br />
<li>The <code>nice</code> command is used to run a command with an adjusted niceness<br />
<ul><br />
<li>generally can’t give negative values to programs without superuser access</li><br />
<li><code>nice -n 12 ./script.sh</code> will add 12 to the default priority (0)</li></ul><br />
</li><br />
<li><code>renice</code> is used to adjust niceness of running processes.<br />
<ul><br />
<li><code>renice 0 -p 2772 3562</code> renices the processes 2772 and 3562 to 0</li><br />
<li><code>renice 1 -u darseife</code> renices the processes of user darsiefe to 1</li></ul><br />
</li></ul><br />
</li><br />
<li>Killing processes<br />
<ol><br />
<li><code>kill</code><br />
<ul><br />
<li>stopping a process currently running <code>-kill -s signal PID</code><br />
<ul><br />
<li><code>PID</code> PID by default, kill sends a signal to a number of process IDs</li><br />
<li>signal 1 is <code>SIGHUP</code> terminates interactive programs and causes daemons to reread config files</li><br />
<li>signal 9 is <code>SIGKILL</code> stops all processes without performing a graceful exit or shutdown</li><br />
<li>signal 15 is <code>SIGTERM</code> asks the process to terminate and to close out open files, run its shutdown script, etc.</li></ul><br />
</li><br />
<li><code>kill</code> will only kill processes owned by the user, unless run with superuser access</li></ul><br />
</li><br />
<li><code>killall</code><br />
<ul><br />
<li>kill multiple things</li><br />
<li><code>killall vi</code> kills all running instances of vi for the user; when run as a superuser, this will kill any running instance of vi</li><br />
<li><code>killall -s 9 vi</code> kills all running instances of vi for the user by sending the <code>SIGKILL</code> signal to it; when run as a superuser, this will kill any running instance of vi.</li><br />
<li><code>killall -i vi</code> uses interactive mode to ask which processes to kill</li><br />
<li>if you are running killall as root, try to use -i to prevent loss of other people’s work</li></ul><br />
</li></ol><br />
</li><br />
<li>Additional process control<br />
<ul><br />
<li>If you are running a program inside of a terminal, <code>[Ctrl + Z]</code> will pause and go to background<br />
<ul><br />
<li>To restart the process, run the command <code>fg</code> in the same terminal</li><br />
<li>With multiple processes paused, <code>fg 1</code> or <code>fg 2</code> will bring back process 1 or 2</li></ul><br />
</li><br />
<li>Processes can be started in the background with &amp; at the end of the line; unlike paused processes, these will continue to run.</li></ul><br />
</li></ol><br />
<br />
<span id="linux-time-and-scheduling"></span><br />
=== Linux Time and Scheduling ===<br />
<br />
# Overview<br />
#* Computer systems in general rely on a pair of clocks to operate<br />
#* Two clocks<br />
#** '''Hardware clock''' - initialized in BIOS/UEFI and kept updated via battery while computer is off<br />
#** '''Software clock''' - initialized at boot from BIOS/UEFI time, and is used to track time while running<br />
#* Typically, the clock of the computer is set using coordinated universal time (UTC)<br />
#** adjusted for use based on timezones and localization settings like DST before entering userspace<br />
#* Both the hardware clock and the software clock are not terribly reliable and tends to drift over time<br />
#* This can be a problem on x86_64 systems in general because many network services rely on accurate timekeeping<br />
#* This is made worse when having hundreds or thousands of servers requesting time from a single source could be bad<br />
# Unix Time Commands<br />
#* <code>date</code> outputs the current software clock time<br />
#* <code>date -u</code> outputs the UTC time<br />
#* <code>date 102715022019</code> sets the time to exactly Oct 27, 15:02, 2019 in local time<br />
#* <code>hwclock</code> used to retrieve the value of the hardware clock; must be done as root<br />
#* setting the hardware clock<br />
#** <code>hwclock --set--date 102715022019</code> is a manual set<br />
#** <code>hwclock -hctosys</code> set the hardware clock to the system clock<br />
#** <code>hwclock --synctohc</code> synchronizes system clock to hardware<br />
#* these are called by daemons and init on startup and shutdown<br />
#* both the system time and the hardware clock still suck and need a replacement<br />
# <code>NTP</code> Network Time Protocol<br />
#* NTP relies on a tree of servers (each level is called a stratum) from a hardware atomic clock server<br />
#* The further down the tree you are the less accurate the time is.<br />
#* One NTP server requesting time from a center will exchange packet via timestamp and the delay between communications is estimated<br />
#* It is best to pick NTP servers with a short delay<br />
#* Typically, a cluster of computer is requested from a local time server, which in turn is the only computer requesting times outside of the network<br />
# <code>timesyncd</code><br />
#* Traditionally, <code>ntpd</code> is the daemon that has provided client time synchronization<br />
#* Newer versions of Ubuntu use a service called <code>timesyncd</code><br />
#** uses a slightly different protocol than NTP, called <code>SNTP</code> (simplified network time protocol)<br />
#** configuration information is at <code>/etc/systemd/timesyncd.conf</code><br />
#** <code>timedatectl</code> is a way to view the current SNTP settings<br />
#* also more information related to this accessible with <code>sudo systemctl status systemd-timesyncd</code><br />
# Running jobs in the future: <code>cron</code><br />
#* <code>cron</code> like timesyncd or ntpd, cron is a daemon that runs continuously and looks for events to cause it to spring into action<br />
#* Most daemons are network services, but cron responds to temporal events<br />
#* A cron daemon wakes up once per minute, and examines its config files in:<br />
#** <code>/var/spool/cron/</code><br />
#** <code>/etc/cron.d/</code><br />
#** <code>/etc/crontab</code><br />
#* Executes command specified:<br />
#** System jobs - run as root, and perform system maintenance tasks (rotate logs, clean out <code>/tmp</code>, remove old files, update system)<br />
#** User jobs - run jobs in the future for users<br />
#* <code>/etc/crontab</code><br />
#** System cronjobs are controlled by this file<br />
#** Normally, begins with environment variable configuration, then several lines like:<br />
#*** On a typical system <code>02 4 * * * root run-parts /etc/cron.daily</code><br />
#*** On Ubuntu Desktop <code>25 6 * * * root test -x /usr/sbin/anacron || cd / &amp;&amp; run-parts --report /etc/cron.daily</code><br />
#** Fields for cron, in order<br />
#**# the minute of the hour<br />
#**# the hour of the day<br />
#**# day of the month<br />
#**# month of the year<br />
#**# day of the week<br />
#** different scripts for items are run from:<br />
#*** <code>/etc/cron.daily</code><br />
#*** <code>/etc/cron.weekly</code><br />
#*** <code>/etc/cron.monthly</code><br />
# <code>anacron</code> a specific implementation of cron<br />
#* <code>anacron</code> is not intended to be a replacement for cron, it is a supplement<br />
#* cron works great on systems that are ''always on''<br />
#* <code>anacron</code> is smart in that it can tell if things are run<br />
#** if things are missed, it plays catchup<br />
#** light-night log rotation, for example, might never run<br />
#** anacron is intended to address this<br />
#*** when run it checks to see when it last executed each program<br />
#*** if a period greater than the programs interval has passed, the program gets run by anacron<br />
#** config file <code>/etc/anacrontab</code><br />
#*** format: <code>1 5 cron.daily run-parts --report /etc/cron.daily</code><br />
#**** first column: delay in minutes<br />
#**** second column: frequency in days<br />
# <code>crontab</code> edit cronjobs for users<br />
#* cron jobs for the system are under <code>/etc/</code> but user ones are kept in <code>/var</code><br />
#* <code>crontab -e</code> edit your user cronfile<br />
#* root can edit other users with <code>crontab -u username -e</code><br />
#* <code>/etc/cron.allow</code> and <code>/etc/cron.deny</code> exist for whitelisting or blacklisting users from using <code>cron</code><br />
#** an empty <code>/etc/cron.deny</code> allows ALL USERS if present<br />
#** an empty <code>/etc/cron.allow</code> blocks ALL USERS if present<br />
#* if neither of these exist, then only root may use <code>cron</code><br />
# <code>at</code> a handy simple task timer<br />
#* cron and anacron can be overkill for simple tasks that need to be done once<br />
#* example:<br />
#** <code>at 5am Oct 31</code><br />
#** <code>at&gt; /home/darren/run-backup.sh</code><br />
#** <code>at&gt; echo &quot;job done&quot;</code><br />
#** <code>at&gt; mail -s &quot;backup job finished&quot; root</code><br />
#** <code>at&gt;</code> '''ctrl+d''' to finish<br />
#* <code>at now +1 hour</code> will run 1 hour from now relative to current system time<br />
#* other examples:<br />
#*# <code>at midnight</code><br />
#*# <code>at noon</code><br />
#*# <code>at teatime</code><br />
#* <code>aterm</code> to remove an upcoming command<br />
#* <code>atq</code> used to list pending commands<br />
<br />
<span id="log-files"></span><br />
=== Log Files ===<br />
<br />
# Intro<br />
#* Linux machines log files that record key details about system operation and are typically stored in <code>/var/log</code><br />
#* Logging on Linux system is generally performed by the syslog daemon, <code>syslogd</code><br />
#** However, some daemons and programs perform logging independently of syslog<br />
#* Logging on Linux can be done locally to the computer, but in a networked environment, logs may be sent to a central logging server and by email<br />
#* Logging is a security action and is typically a part of the system worth hardening because it is exploitable by an attacker to hide their break-in<br />
#* Logs should be rotated, aka moved out of the system, to prevent <code>/var/log</code> from being filled completely<br />
# <code>syslog</code><br />
#* The basic idea of a system logger is to provide a unified means of a handling log files<br />
#* The daemon runs in the background and and accepts data delivered from servers and other programs<br />
#* Once the log daemon receives a message, it is classified and directed to an appropriate file<br />
#* <code>syslogd</code> might be managing several services at the same time and providing different rule-based logging based on priority of the messages received<br />
#* <code>syslogd</code> relies on the config file <code>/etc/syslog.conf</code><br />
#** Ubuntu uses <code>/etc/rsyslog.conf</code> for rocket-fast syslog<br />
#** the configuration file supports a ruleset, located at <code>/etc/rsyslog.d/</code> or <code>/etc/syslog.d/</code> respectively<br />
#** standard logging is done by daemon, with the format <code>facility.priority action</code><br />
#*** example: <code>mail.* /var/log/mail.log</code><br />
#** facility parameter is the name of the type of program the daemon that generates messages to be logged<br />
#**# <code>auth/priv</code> security messages<br />
#**# <code>cron</code> from the cron daemon<br />
#**# <code>daemon</code> catch-all for various systems servers<br />
#**# <code>kern</code> kernel messages<br />
#**# <code>lpr</code> printer and CUPS messages<br />
#**# <code>mail</code> mail subsystems<br />
#**# <code>news</code> oldschool newsgroups<br />
#**# <code>syslog</code> internally generated messages<br />
#**# <code>uucp</code> unix-to-unix copy daemon<br />
#**# <code>local[0-7]</code> predefined facilities for custom configurations<br />
#** priority parameters include:<br />
#*** <code>deb</code> debugging messages<br />
#*** <code>info</code> normal operations<br />
#*** <code>notice</code> abnormalities that do not effect functionality<br />
#*** <code>warn</code> tells that an error will occur is action is not taken<br />
#*** <code>err</code> standard error for the daemon, non-urgent failure of the daemon causing an error message<br />
#*** <code>crit</code> critical error causing a secondary system to fail urgently<br />
#*** <code>alert</code> critical error causing a primary system to fail urgently<br />
#*** <code>emerg</code> Panic message - system unusable<br />
#** the action parameter is a file or location on the network that will accept the message<br />
#*** could be a file in <code>/var/log</code><br />
#*** could also be a tty: <code>/dev/tty8</code><br />
#*** could also be a computer on the network: <code>@logger.ndus.edu</code><br />
#* When a program sends a message to the system logger, it includes the priority: the message is only logged if it is of a certain level or higher<br />
#** Exceptions:<br />
#*** <code>*</code> all codes<br />
#*** = <code>(mail.=crit)</code> applies to this priority only<br />
#** You can also specify multiple facilities with the same priority by using a comma<br />
#** Multiple facilities separated by commas followed by a priority all follow the same priority<br />
#** A semicolon is used to specify a number of logging priorities <code>kern.info;kern.err</code> everything between info and error<br />
# <code>logger</code> - logging something else<br />
#* The <code>logger</code> command allows a user to log something else other than established facilities<br />
#* <code>logger shutting down for system maintenance</code><br />
#** will log to <code>/var/log/syslog</code> as <code>Nov 6 15:50:00 darseife-virtualbox darseife: shutting down for system maintenance</code><br />
#* <code>-i</code> records the PID of the logger process when run (useful when run from script)<br />
#* <code>-s</code> print the log message to standard error in addition to logging<br />
#* <code>-f /foo</code> logs to <code>/foo</code> instead of the system log<br />
#* <code>-p</code> specify both a facility and a priority for the system log message; <code>-p local3.info</code><br />
<br />
<span id="firewalling"></span><br />
=== Firewalling ===<br />
<br />
# TCP Wrappers<br />
#* TCP wrappers depend on two config files to work properly; these are <code>/etc/hosts.allow</code> and <code>/etc/hosts.deny</code><br />
#* the most common config for these is default <code>deny</code><br />
#** <code>ALL:ALL</code> in <code>/etc/hosts.deny</code><br />
#** <code>ALL:127.0 [::1]</code> in <code>/etc/hosts.allow</code><br />
#** specific services from a specific host enabled as the form <code>sshd:192.168.X.Y</code> in <code>/etc/hosts.allow</code><br />
#** all services from a domain as <code>ALL:.minotstateu.edu</code> in <code>/etc/hosts.allow</code><br />
#* Also used is specific allow<br />
#** nothing is listed in <code>/etc/hosts.allow</code><br />
#** specifically removed IP addresses in the form <code>ALL:165.234.X.Y</code> in <code>/etc/hosts.deny</code><br />
#* Secondary service-level blocking can be used to apply rules to services in the form <code>/etc/sshd.hosts</code><br />
# <code>iptables</code><br />
#* <code>iptables</code> is the kernel level netfilter firewall<br />
#* <code>iptables -L</code> lists the current ruleset<br />
#* <code>iptables -A</code> is used to APPEND new rules to a chain (takes a chain name)<br />
#* <code>iptables -I</code> inserts a rule (takes a chain name and a position)<br />
#** example <code>iptables -A INPUT -ptcp -dport ssh -j ACCEPT</code><br />
#* rule options<br />
#** <code>-p</code> the protocol used<br />
#** <code>-d</code> the destination port, can accept a range of port<br />
#** <code>-j</code> the action to take, includes ACCEPT, DENY, DROP<br />
#** <code>-s</code> specify a source address, can be either <code>192.168.0.1</code> or <code>192.168.0.0/24</code><br />
#** <code>-m</code> require the rule to match a specific property<br />
#*** ex <code>iptables -m conntrack</code> allows filter rule to match based on connection state<br />
#** <code>--ctstate</code> define a list of connection states for the rule to match on<br />
#**# NEW - new connections<br />
#**# RELATED - new but related to a previous on<br />
#**# ESTABLISHED - a preexisting connection already in progress<br />
#**# INVALID - any malformed traffic or unidentified traffic<br />
#** <code>-j</code> tells iptable what to do with the packet given the rule<br />
#*** ACCEPT - routes as normal as per IP header<br />
#*** DROP - kills the packet without giving reply<br />
#*** REJECT - sends an explicit error message<br />
# Uncomplicated Firewall (ufw)<br />
#* <code>ufw enable</code> to turn on ufw<br />
#* <code>ufw disable</code> to turn off ufw<br />
#* <code>ufw status</code> view current rules for ufw<br />
#* <code>ufw status verbose</code> reports logging level + access levels for each rule<br />
#* <code>ufw allow 80/tcp</code> sets a rule to allow tcp packets on port 80<br />
#* <code>ufw deny 22/udp</code> sets a rule to reject udp packets on port 22<br />
#* <code>ufw deny ssh</code> sets a rule to reject all ssh protocol packets<br />
#* <code>ufw allow 6000-6007/tcp</code> sets a rule with a range to allow tcp packets from 6000-6007<br />
#* <code>ufw allow from 15.15.15.1 to any</code> sets a rule allowing all traffic from 15.15.15.1<br />
#* <code>ufw status numbered</code> returns a numbered list of rules in the set<br />
#* <code>ufw delete 2</code> removes rule number 2 in a list<br />
#* <code>ufw logging {on/off}</code> turns the logging on or off; usually caught by syslogd/rsyslog; logging is set to <code>low</code> by default<br />
#* <code>ufw logging {low/medium/high/full</code> sets logging verbosity of ufw;<br />
#** note that logging is rate limited at <code>medium</code> and not rate limited at <code>high</code><br />
<br />
<span id="bootinitialization"></span><br />
=== Boot/Initialization ===<br />
<br />
# Slackware - SysV Init<br />
#* System 5 style rc script via <code>init</code><br />
#* <code>PID 1</code> = <code>init</code> which is the process that starts all other processes when the system boots<br />
#* <code>init</code> is responsible for processing <code>/etc/inittab</code> to see how to start the system and at what runlevel<br />
#* Things <code>init</code> does<br />
#** Runs the script <code>/etc/rc.d/rc.s</code> to determine to prepare the system before loading to the desired runlevel<br />
#** Enables virtual memory for the system and mounts filesystems in <code>/etc/fstab</code><br />
#** Loads kernel modules<br />
#** Near the end of the script <code>rc.s</code>, init will run scripts related to the runlevel the system is entering -for example, <code>rc3.d</code> is run at runlevel 3<br />
# SystemD<br />
#* Ubuntu, CentOS, RedHat, and others have moved from SysV init process to the SystemD initialization process<br />
#* Rather than running many shell scripts during init, the systemD process uses one monolithic program to control startup of services<br />
#* The way the services are started are controlled by a configuration file<br />
#* systemd uses ''units'' and ''targets'' rather than scripts and runlevels<br />
#** there are 11 different types of units:<br />
#*** <code>mount</code> unit defines a mountpoint on the system named with the mount path with slashes replaced as dashes<br />
#*** <code>swapspace</code><br />
#*** <code>device</code> a device in the linux device tree<br />
#*** <code>socket</code> fifo pipe to an internet port with an associated service<br />
#*** <code>service</code> a service or application on the system<br />
#** targets are used to stop and start the service and group units together<br />
#** systemd identifies units by name and type<br />
#*** example ssh could have an ssh.service and an ssh.socket<br />
#* <code>systemctl</code> - can list current units loaded on the system<br />
#* config files - <code>/lib/systemd/system</code><br />
#* modify how a unit functions - <code>/etc/systemd/system</code></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Unix_crash_course&diff=131Unix crash course2022-10-21T18:02:56Z<p>Sysadmin: </p>
<hr />
<div><br />
==Introduction to Unix==<br />
<br />
===Syllabus===<br />
<br />
<ol style="list-style-type: decimal;"><br />
<li>Recommended material:<br />
<ul><br />
<li>Unix and Linux System Administration 5th Edition - Evi Nemeth, et al.</li><br />
<li> CompTIA Linux+ Study Guide 4th Edition - Chritine Bresnahan and Richard Blum</li></ul><br />
</li><br />
<li>Objectives<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>Shell scripting, input redirection, and C++/C code in a GNU/Linux environment</li><br />
<li>Understanding of Linux file system including permissions</li><br />
<li>Understanding of command and how to intermingle them to achieve more complex goals </li><br />
<li> Basic system architecture, installation process, and management of Linux/Unix</li> </ol><br />
</li><br />
<li>General notes<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>Learning to use Bash scripting and <code>vi</code>/<code>neovim</code> essential to professional usage of Linux in real life</li><br />
<li> Any commands discussed in class can be looked up via ‘man’</li><li>More detailed information for any command available from the command shell with an internet connection: <code>curl cheat.sh/command_na</code></li></ol><br />
</li></ol><br />
<br />
===History===<br />
<br />
<li> Bell System <br />
#<li>Dates back to late 1960s, Originally developed by Ken Thompson and Dennis Ritchie of Bell Labs </li><li>Ritchie described the initial goal as to create a good environment in which to: <br />
##<li>do programming and development </li><li>foster communal computing through remote access and close communication </li></li><li>Unix originally written in assembly, but was later re-written in C in 1972 </li><li>First public releases in 1973, at Symposium on Operating Systems Principles; early source code sent by standard mail </li><li>1975, Version 6 of Unix was licensed to companies for the first time. However, commercial users were rare. Licensees only received source code from AT&amp;T. </li><li>Also in 1975, Ken Thompson takes a sabbatical from Bell Labs as a professor at UC Berkeley. <br />
##<li>Creates BSD: The Berkeley System Distribution of Unix 4.2. </li><li>Includes C-shell and vi editor for first time </li></li><li>1983, Bell broken into several companies in anti-trust suit against AT&amp;T. </li><li> AT&amp;T new restrictive license almost kills commercial Unix. <br />
##<li>By 1991, BSD had grown to a complete rewrite of AT&amp;T Unix and comes with BSD license. </li><li>BSD survives today as: FreeBSD, OpenBSD, NetBSD, etc. </li><li>Additionally, Darwin kernel of Mac OSX continues as a form of Unix. </li></li></li><li>GNU Project into Linux <br />
#<li>Around 1983, Richard Stallman creates GNU project </li><li>Stallman’s GNU system goal was to create a ‘free’ system of software that had many of other capabilities of the commercial tools that many Unix also had </li><li>By 1985, Stallman had founded the Free Software Foundation to raise funds to develop free versions of all the components of the Unix syste, under GNU GPL </li><li>In 1991, Linus Torvalds began the development of a new free OS based on the MINIX operation system and SunOS. The 21 year old puts together the kernel. </li><li>As the kernel for the GNU project was still very much in development, the two projects became a natural fit. </li><li>To this day, Stallman insists that Linux is GNU/Linux due to the large reliance on GNU toolchain </li></li><br />
<br />
===Introduction to Red Hat Enterprise Linux=== <br />
#Package Manager - YUM <br />
##Stands for YellowDog Updater Modified<br />
##Provides a set of tools for retrieving software from the internet using subscriptions to an update server<br />
##Utilizes .rpm software files to install packages for management<br />
##Creates an Enterprise Linux environment <br />
##*'''cons:''' less software options overall<br />
##*'''pro:''' all available software is tested and works without additional configuration<br />
##5. <code>yum</code> has an additional, alternative wrapper program <code>dnf</code> to improve output and ease of use <br />
##*stands for DaNdified YUM - commands in the format <code>sudo dnf [verb] [objects]</code><br />
##*<code>-y</code> flag gives YES as passed option for less attended installation 6. command commands<br />
##*<code>sudo dnf -y install [package_name]</code> install a particular package and dependencies<br />
##* <code>sudo dnf -y update</code> update all packages<br />
<br />
===Introduction to Ubuntu===<br />
# Package Manager - APT<br />
##Stands for Advanced Package Manager<br />
##Ubuntu’s set of tools to install packages and keep them up to date<br />
##Utilizes .deb software packages for management<br />
##Keep in mind this package management system is far from standard <br />
##*Many commercial Unixes have their own package managers<br />
##*When all else fails, a tarball (.tar) file can be used to build from source<br />
##Commands <br />
##*<code>apt-get update</code> synchronizes the index of packages to your system<br />
##*<code>apt-get upgrade</code> downloads and installs new versions of all currently installed packages in place, based on the index<br />
##*<code>apt-get dist-upgrade</code> downloads and installs new versions, and moves the system up to the next version, based on the index<br />
##*<code>sudo</code> was originally “superuserdo”, now works to substitute - users and run programs as that user. /etc/sudoers controls this behavior.<br />
##*<code>sudo apt-get install foo</code> : installs package named '''foo'''<br />
##*<code>sudo apt-get remove bar</code> : removes a package named '''bar''', but not the config files<br />
##*<code>sudo apt-get purge bar</code> : removes a package named '''bar''', and removes config files<br />
##*<code>sudo apt-get source * </code>: gets the source code for a package<br />
##*<code>sudo apt-get clean</code> : removes downloaded packages (.deb files) used to install<br />
<span id="filesystem"></span><br />
===File System===<br />
<br />
<ol style="list-style-type: decimal;"> <br />
<li>Directories<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>first and bottom of the directory of the file structure tree is called root '''/'''</li><br />
<li>'''/root''' in addition, there is a file directory, within the root directory called /root (“slash root”), which acts as root’s home directory </li><br />
<li>'''/home''' normal users’ directories</li><br />
<li>'''/boot''' contains the boot directory, including kernel, initial RAMdisk, and bootloader</li><br />
<li> '''/dev''' lists all devices of the computer</li><br />
<li>'''/etc''' configuration files for nearly all programs that need a state</li><br />
<li>'''/tmp''' temporary files for the system</li><br />
<li>'''/var''' variable-length files used by programs<br />
<ul><br />
<li>'''/var/spool''' old-school webmail location</li><br />
<li>'''/var/www''' location of http server files</li></ul><br />
</li><br />
<li>'''/usr''' files available to users, including binaries for daily user tasks in /usr/bin </li><br />
<li>'''/sbin''' binaries used only by root</li><br />
<li>'''/bin''' binaries for the system everyone needs to use</li></ol><br />
</li><br />
<li>Commands<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>'''cd''' change directories</li><br />
<li>'''~''' alias used to shortcut the current user directory</li><br />
<li>'''Ctrl+R''' search history of commands in <code>bash</code>; also available as output from <code>history</code></li><br />
<li>'''!!''' runs the previous command</li><br />
<li>'''ls''' list in the contents of current directory</li><br />
<li>'''.''': the current directory</li><br />
<li>'''..''' : go up one directory#User and Group Configuration</li></ol><br />
</li></ol><br />
===useradd/usermod=== <br />
<br />
#<code>useradd</code> and <code>usermod</code> do much the same thing<br />
#* <code>-c</code> allows to add comments<br />
#*<code>-d</code> allows to change user home directory<br />
#*<code>-e</code> expiration date - date an account will expire (in YYYYMMDD format), kept in <code>/etc/shadow</code><br />
#*<code>-f</code> specify how many days the account can be inactive before password expires, kept in <code>/etc/shadow</code><br />
#*<code>-g</code> specify default group of the user<br />
#*<code>-G</code> append a list of additional groups for the user<br />
#*<code>-l</code>(usermod only) changes a user login ex: <code>usermod -l darren darseife</code><br />
#*<code>-u</code> uid<br />
#*<code>-s</code> specifies login shell<br />
#*<code>-p</code> specify pre-hashed password<br />
#*<code>-L</code> (usermod only) locks the user’s password so it cannot be entered directly (best used with -e, as well)<br />
#*<code>-U</code> will unlock a user’s password<br />
<br />
===passwd===<br />
commands<br />
<br />
# <code>sudo passwd darseife</code> change the password of darseife<br />
#<code>passwd</code> can also be used to lock and unlock accounts<br />
#*<code>passwd -l</code> will lock the account’s password<br />
#*a locked password will always be input incorrectly<br />
#*does not necessarily lock user out of the account<br />
#*puts an <code>!</code> at the beginning of the password field in <code>/etc/shadow</code><br />
#*<code>crypt</code>, the algorithm used to implement ciphers, cannot generate an <code>!</code>so the password hash is disabled<br />
#*<code>passwd -u</code> would unlock the account’s password<br />
#<code>passwd -d</code> removes the password of the account<br />
#<code>passwd -S</code> displays account information: <br />
#*last password change<br />
#*password expires<br />
#*password inactive<br />
#*account expires<br />
#*minimum days between password change<br />
#*maximum days before password change<br />
#*warn period before changes<br />
<br />
===chage===<br />
1. The <code>chage</code> command allows you to modify account settings related to password expiration and aging 2. <code>chage</code> can also be used to configure accounts to automatically expire if: - the password hasn’t been changed - or a date has been passed 3. <code>chage -l</code> print expiration settings of an account 4. <code>chage -m DAYS username</code> to set the minimum time between password changes 5. <code>chage -M DAYS username</code> to set the maximum time between password changes 6. <code>chage -d YYYYMMDD username</code> update the last day YYYY-MM-DD on which to change the password 7. <code>chage -E YYYYMMDD username</code> set the expiration date as YYYY-MM-DD on which username’s account expires - setting the number to <code>-1</code> will make it never expire 8. <code>chage -I DAYS username</code> will lock the account password after number of days inactive 9. <code>chage -W DAYS username</code> number of days the user will be warned before their account is made inactive 10. All of these settings are stored in <code>/etc/shadow</code><br />
<br />
###userdel 1. <code>userdel username</code> deletes users 2. <code>userdel -r username</code> deletes the user, their mail spool, their home directory, and everything in it 3. <code>userdel -f username</code> deletes the user ''right now'' even if they are logged in<br />
<br />
###groupadd and groupmod 1. <code>groupadd</code>add a group to a system 2. <code>groupadd -r</code> used to create a system group, creating a GUID above 1000 3. <code>groupmod</code> change a group 4. <code>groupmod -n</code> change a group name<br />
<br />
###gpasswd - group password 1. <code>gpasswd groupname</code> sets password for a group; it is most common to not have a group password 2. <code>gpasswd -R</code> restricts access to users logging in with password via newgrp<br />
<br />
###getent - get entities 1. can be used to view individual records in <code>/etc/passwd</code>, and other files 2. useful for getting exact matches for items 3. <code>getent password username</code> - get passwd information for user 4. <code>getent shadow username</code> - get shadow information for user 5. <code>getent group groupname</code> - get group information for user#The *nix CLI ###The Shell explained 1. A shell is a command interpreter where the user interacts with the system via command line interface (CLI) 2. Different shells currently in-use A. '''Bourne-Again SHell''' - located at <code>/bin/bash</code> is the main shell and is used was a default on most Linux systems B. '''sh''' - The Bourne Shell, old school version does not require any libraries. Rarely seen as a user-interface shell; typically a symbolic link to another shell binary. Created by Stephen Bourne, first included in Unix V7 C. '''tcsh''' Tee Shell. Based on the earlier C-shell from Unix. Has some additional features built into it. D. '''csh''' C-shell, a BSD component not often seen on Linux. Created by Bill Joy at UC Berkeley. E. '''ksh''' Korn shell from Bell Labs in the 1980s. The idea was to incorporate features from the Bourne Shell along with C-programming syntax. F. '''zsh''' 1990s, incorporates additional ideas into ksh. 3. Built-in Commands A. <code>cd</code> changes directory - <code>cd ~</code> or <code>cd /home/user</code> are equivalent B. <code>ls -</code> lists files in a directory - default behavior just lists files - <code>-a</code> lists all files including hidden ones - <code>-l</code> long format - <code>-d</code> just directories - <code>-F</code> uses a special character to rep filetypes - <code>-R</code> recursive listing C. Single characters are substituted by <code>?</code>, wildcards by <code>*</code> D. Exiting the system - <code>exit</code> exits the current shell - <code>logout</code> only works for login shells E. Shutting down the system - <code>shutdown -h now</code> halts the system now - <code>shutdown -r now</code> restarts the system now - <code>shutdown -h +5 &quot; &quot;</code> broadcasts the message in quotes and tells people it will be down in 5minutes - <code>halt</code> on most systems, will drop the system into single user mode - <code>poweroff</code> “on fire” graceless exit. Sends HW signal to kill power - <code>reboot</code> essentially ‘shutdown -r’ - <code>reboot -f</code> graceless restart exit, Sends HW signal to reboot F. <code>init</code> relies on a series of programs or scripts to change run-state of machine and to start/stop linux services - <code>init 0</code> - shuts the machine down - <code>init 6</code> - reboots the machine - <code>init S</code> - initializes system boot <br />
<br />
###Man pages and getting help 1. displays the manual pages for any packages on the system 2. typing <code>[/]</code> will allow you to search through the page 3. allows you to move via <code>[spacebar]</code> or <code>[pgup/pgdn]</code> 4. <code>man</code> is broken up into seperate chapters: <code>man 5</code> would show chapter 5 1. executable programs and shell 2. system calls 3. library calls 4. device calls 5. file formats 6. games 7. misc. (macro packages) 8. sys admin commands 9. kernel routines 5. <code>man -k *</code> : to find any pages matching <code>*</code> 6. most programs also allow for <code>--help</code> or <code>-?</code> to be passed for a quick rundown on commands and usage<br />
<br />
###Unix filesystem permissions 1. filesystem permissions saved as a series of 3 bit numbers 2. format: <code>d,rwx,rwx,rwx</code> for read-write-execute a. <code>d</code> - denotes a Directory or file b. first triad is for the User of the file c. second triad is for Group members of the file d. third triad is for Everyone else 3. Read-write-execute for files a. '''Read''' allows the file to be opened b. '''Write''' allows for the file to be edited c. '''Execute''' allows for the file to run as a program 4. Read-write-execute for directories a. '''Read''' allows for the directory to view filenames in the directory b. '''Write''' allows files to be added or deleted from a dir c. '''Execute''' allows the directory to be visited, “permission to traverse through”; must have this to execute programs within a directory 5. <code>chmod</code> changes file and directory permissions a. two modes for chmod 1. '''symbolic mode''' <code>chmod {u,g,o}[+,-,=]{r,w,x}</code> 2. absolute mode - chmod 777 /path/to/file` * 0 no permissions * 1 execute * 2 write * 4 read F. chown - change file owner and group 1. chown user /foo - changes foo’s owner to user 2. chown root:root /foo - changes foo’s owner to root, also changes foo’s group to root G. echo - displays any text you enter after the command 1. primarially used in shell scripts or ‘programs’ that utilize a combination of of built-in commands, variables, and executables to complete tasks 2. variables reached from echo by invoking the variable name as $NAME #Input Redirection and Utils <br />
<br />
###Redirection symbols 1. prog &gt; - redirects std output to the file, truncating and rewriting 2. prog 2&gt; - redirect std error to the file, truncating and rewriting 3. &amp;&gt; - redirects both std output and error to file 4. <code>&gt;&gt;</code> - non-destructively appends to end of file rather than rewriting 5. <code>- &quot;backtick&quot; process commands in the middle of another command a. A tool similar to xargs b. text within backticks are treated as a separate command line who's results are substituted on this command line c. things inside of backticks are attempted to be used as a command d. example: rm</code> find / - user darren ` 6. $() - is used like a backtick; more resistant to weird syntax<br />
<br />
###Piping data between programs 1. programs on a UNIX system frequently use pipes to redirect each other’s output. 2. for instance you might use a text-filtering command on output from a file reading command. 3. allows small programs to operate on a flow of information in a row 4. | - this is an unnamed pipe 5. mkfifo - allows for the creation of named pipes a. <code>mkfifo fifo1</code> b. <code>ls -l &gt; fifo1</code> c. <code>cat &lt; fifo1</code> 6. tee - read from standard input and write to standard output and files a. splits standard input so that is it is both displayed via standard output and also captured to one of more files b. allows for logging and redirection while the process is also shown on screen c. overwrites to a file as per ‘&gt;’ by default rather than like ‘&gt;&gt;’ d. example - echo $PATH | tee file1 file2 - file1, file2, and stdoutput all have same information 5. find - searches for files in directory hierarchy 6. xargs - builds a command from standard input, using command name and arguments to that command and then executes the command on each line of standard input a. example: find / - user darren | xargs -d %rm 7. file - returns the type of file and metadata information regarding files 8. join - combines two text files by matching the contents of specified fields a. by default, join uses spaces as the limiters (unless -t CHAR is given) b. example: join -1 3 -2 2 - this joins the third field of the first file and the second field of the second file 9. paste A. Paste is like cat rotated 90 degrees. When operating on multiple files, it will merge the lines of the files. 10. expand - converts tabs to spaces A. by default assumes you want 8 spaces per tab; -t changes B. unexpand - converts spaces to tabs 11. od - dump files into octal and other formats A. default usage is dumping binary file into octal binary format B. -x for 2byte hex output 12. sort - sorts lines of text files A. writes sorted concatenation of all files as output B. by default, uses first column C. Important options 1. -f : ignore case 2. -m : sort by three digit month abbreviation; ex, jan 3. -n : numeric sort 4. -v : sort by versio 5. -h : compare human readable numbers 6. -r : reverses the sort 7. -R : sorts randomly but groups like files 8. -d : dictionary-order 9. -k : sort via key, sorts by a field number, can be comma-seperated listq D. the opposite of this is ‘shuf’ 13. split - splits a file into 2 or more files A. needs to have an output file prefix B. where do you want to split 1. -b : size in bytes 2. -l : lines C. example: split -l 2 listening1.txt numbers -&gt; output is: ‘numbersaa’ and ‘numbersab’ D. by default: splits every 10,000 lines 14. tr - trnaslate or delete characters A. changes individual characters coming in through standard input and outputs via standard output B. options 1. [:lower:] - all lower case chars 2. [:upper:] - all upper case chars 3. [::] 15. cut - extracts portions of lines of input and prints out as output A. used to remove items within lines, delimited by spaces or tabs B. options 1. -b : cuts the specified list by byte 2. -c : cuts the specified characters 3. -f : cuts specified list of fields from input 4. -d : specifies the delimiter for fields C. EXAMPLE: cut -f 7 -d : /etc/passwd 16. uniq - take an input list and report or omit repeated lines A. only if they are next to each other, so you have to sort first B. EXAMPLE: cut -f 7 -d : /etc/passwd | sort | uniq<br />
<br />
<span id="c-programs-and-makefiles"></span><br />
===C++ programs and Makefiles===<br />
<br />
#Basic C++ program with input and output<br />
<br />
<pre>int main (int args, char * argv[]){<br />
for(int x = 0; x &lt; argv[],x++){<br />
cout &lt;&lt; arv[x] &lt;&lt; endl;<br />
}<br />
return 0;<br />
}<br />
// outputs every character given to it</pre><br />
<ol start="2" style="list-style-type: decimal;"> <br />
<li>Let’s write a less good version of grep called greb:</li></ol><br />
<br />
<pre>#include &lt;iostream&gt;<br />
#include &lt;fstream&gt;<br />
using namespace std;<br />
<br />
int main(int argc, char * argv[]) {<br />
string curLine;<br />
if (argc == 2){<br />
while(getline(cin,curLine)) {<br />
if (curLine.find(argv[1]) != string::npos) {<br />
cout &lt;&lt; curLine &lt;&lt; endl;<br />
}<br />
}<br />
}<br />
<br />
else if (argc == 3){<br />
ifstream inFile(argv[2], ifstream::in);<br />
<br />
while(getline(inFile,curLine)) {<br />
if (curLine.find(argv[1]) != string::npos) {<br />
cout &lt;&lt; curLine &lt;&lt; endl;<br />
}<br />
}<br />
}<br />
else {<br />
cerr &lt;&lt; &quot;Error: unknown parameters.&quot; &lt;&lt; endl;<br />
return 1;<br />
}<br />
return 0;<br />
}<br />
</pre><br />
<ol start="3" style="list-style-type: decimal;"><br />
<li>Make A. Allows for automatic configuration and creation of usable binararies from OOP files, and can be used to install and uninstall programs B. Allows for the creation of archives of source code that build as packages into useful code C. typically named ‘makefile’</li> </ol><br />
<br />
<pre># set a variables for the compiler<br />
CC = g++<br />
LC_MESSAGES=C<br />
CFLAGS=-g -Wall<br />
<br />
# target name variable<br />
TARGET = greb<br />
<br />
# file extension<br />
FILEXT = cpp<br />
<br />
# the build target executable =&gt; install path of the executable<br />
PREFIX = /usr/local<br />
DESTDIR = /home/bernd/staging/<br />
<br />
# basic makefile format<br />
# name of instruction: dependencies<br />
# &lt;tab&gt; command to execute<br />
# the FIRST one you list is the default<br />
$(TARGET): $(TARGET).$(FILEXT)<br />
$(CC) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)<br />
<br />
.PHONY: all<br />
all: $(TARGET).$(FILEXT)<br />
$(CC) $(MAKEOPTS) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)<br />
<br />
.PHONY: clean<br />
clean:<br />
rm $(TARGET)<br />
<br />
.PHONY: install<br />
install: $(TARGET)<br />
mkdir -p $(DESTDIR)$(PREFIX)/bin<br />
cp $&lt; $(DESTDIR)$(PREFIX)/bin/$&lt;</pre><br />
<span id="tar-and-packaging"></span><br />
=== Tar and Packaging===<br />
<br />
#tar A. the name ‘tar’ comes from its original purpose: Tape ARchive B. you can bundle files as .tar, ‘tarballs’, which contain a series of file entries and terminating bytes between files C. each entry has a file descriptor and the binary contents of the file D. file descriptor, or header, contains the name of a file, a checksum of that file, and permissions data E. does not compress data, but instead keeps it in one file F. tar options: -c: Creates a tar archive -v: Verbose output -x: eXtract from a tar archive -f: specify a Filename -A: Appends two archives together -M: create or extract a multivolume archive -j: specify bzip2 compression -J: specify xz compression -z: specify gzip compression –lzma: specify lzma compression -t: Test an archive<br />
<br />
<span id="more-unix-commands"></span><br />
===More Unix Commands===<br />
<br />
#fmt - format A. some files have crazy long lines of text B. reformats to a certain -width C. by default, truncates after 75 characters and cleans up paragraphs D. anything with 2+ blankline removed, and adds indention E. -s : prevents truncation of paragraphs<br />
#nl - line number A. nl does count whitespace by default B. identical to cat -b (but not cat -n)<br />
#pr - prepare a file for printing A. includes header, footer, and page breaks B. create colummns: pr -3 file.txt C. -l sets the length of lines D. -o choose the header text<br />
#regular expressions - sed and awk A. simliar to wildcard expansion: ex; ls ''.txt B. grep and sed are two examples of regEx programs C. unix system programs have two different forms of regular expressions: basic and extended D. types of expresssions: 1. exact text match: grep “blah” returns all lines with “blah” 2. bracketed expressions: b[aeiou]g - returns bag, beg, big, bog, bug 3. bracketed with range: a[2-4]z - returns a2z, a3z, a4z 4. matching single char: a.z (allows for anything between a and z) 5. matching begin/end of line: ^ beginning, $ end 6. any of these can be combined with the wildcard: '' ex: A.* Lincoln (with 0 or more occurances of a pattern.) ex: A[ae]* Lincoln (with 0 or more occurances of a-e after A. 7. matching at least one of: + symbol ex. A[ae]+ Lincoln - returns at least Aa or Ae Lincoln 8. multiple possibilities or: | needs to be set up with ’ ex. ‘Darren (F|Fredrick) Seifert’ 9. grep -E invokes regex functionality<br />
<br />
<span id="shell-scripting"></span><br />
===Shell Scripting===<br />
<br />
#The first line of a shell script tells which program to use to interpret the contents: #!/bin/bash or another shell A. the first two characters are (#!) are special code to tell the kernel this is a script - crunchbang, shebang, hashbang, poundbang B. the remaining lines beginning with # are considered comments <code># this is a comment</code><br />
#Classic Hello World<br />
<br />
<pre># hello.sh<br />
#!/bin/bash<br />
# Print out hello world<br />
echo 'Hello World!'</pre><br />
<ol start="3" style="list-style-type: decimal;"> <br />
<li>When you want to run a shell script, it needs to be made executable before it is an executable file: ‘# chmod u+x ./file.sh’</li><br />
<li>Can be invoked with ./file.sh</li><br />
<li>Alternative routes: A. bash file.sh works whether or not #!/bin/bash or chmod +x has been given B. exec file.sh is used to run scripts from within other scripts and to stop the tty instance after it is done (used to run scripts within scripts)</li><br />
<li>Passing Parameters A. much like C++ programs, bash scripts can be passed parameters B. accessing these, the dollar sign is used, much like PHP or Perl, to call the number of the parameter 1. $0 - the name of the running script itself 2. $1 - the first parameter 3. and so on… 4. $? - returns the exit status of the last run command 5. $$ - returns the process ID (PID) of the current script 6. $USER - returns the username of the user running the script 7. $SECONDS - returns the runtime of the current script 8. $RANDOM - generates a random number (int 0 - 32000) 9. $LINENO - current line of script C. creation of new variables 1. variablename=value # no spaces ever 2. if you do want spaces, they must be escaped by quotes 3. double quotes recommended for variable substitution later 4. for instance varname=“value $1” allows for insertion of other variables into variables whereas varname=’value <math display="inline">1' will not work D. saving output into variables 1. output from programs can be saved into a variable 2. example: curdir=</math>(pwd) will run pwd and return the standard output as a variable 3. if you want to capture the standard error for a command, it can be saved as <code>error=$(&lt;/tmp/error)</code> 4. alternative use: curdir=<code>pwd</code> note the backtics</li></ol><br />
<br />
<pre># foo.sh<br />
#!/bin/bash<br />
# An example script<br />
cp $1 $2<br />
#verify it worked<br />
echo &quot;Details for $2&quot;<br />
ls -l $2</pre><br />
*handling user input and output can use ‘echo’ to create interactive scripts:<br />
<br />
<pre># plususer.sh<br />
#!/bin/bash<br />
echo -n 'Please enter user name for account:'<br />
read username # this saves the stdin input as username<br />
read -p 'Username' $username # -n no-newline<br />
echo $username<br />
read -sp 'Password:' password # -s stealth</pre><br />
*scripts can also handle files as standard input like other commands<br />
<br />
<pre># salesreport.sh<br />
#!/bin/bash<br />
#a sales report generated from standard input<br />
echo Data Summary:<br />
cat /dev/stdin | cut -d ' ' -F2,3 | sort</pre><br />
*variables can be made global via export<br />
<br />
<pre># script.sh<br />
#!/bin/bash<br />
var1=blah<br />
var2=foo<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
export var1<br />
exec ./script2<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre><br />
<pre>script2.sh<br />
#!/bin/bash<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
var1=flop<br />
var2=bleh<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre><br />
<span id="sed---stream-editor"></span><br />
===Sed - Stream EDitor=== <br />
<br />
#the sed command uses regular expressions to modify stdin and sends the changes to stdout<br />
#usage : <code>sed [options] script.txt [inputfile]</code><br />
#some of the commands that can be used in the script portion <br />
#* <code>=</code> - display the current line number<br />
#*<code>a\text</code> - append text to file<br />
#*<code>i\text</code> - insert text to file<br />
#*<code>r file</code> - append text from filename into the file<br />
#*<code>c\text</code> - replace the selected range with the text<br />
#* <code>s/2012/2013/</code> - replace the first occurrence of 2012 on first occurrence on each line<br />
#* <code>s/2012/2013/g</code> - replace ’’ ’’ every occurrence on each line<br />
#examples<br />
#*<code>sed 's/Nick\|nick/Gary/g' &lt; salaryprojections.txt</code><br />
#*^ this would replace any ‘Nick’ or ‘nick’ with ‘Gary’ at all points in the file<br />
#*standard replace format: <code>sed 's/^/ /' file.txt &gt; file.new.txt</code> ### Examining Processes<br />
#<code>ps</code> command<br />
#*The internal Unix command for process control is <code>ps</code><br />
#*By default, <code>ps</code> reports back what is happening inside of the current terminal/tty;<br />
#* shows <code>PID</code>, <code>TTY</code>, <code>TIME</code>, and <code>CMD</code><br />
#**<code>PID</code> is the Process ID number for a running program<br />
#**<code>TTY</code> is the Teletypewriter device which the process runs from; reports stdout and stderr to this by default<br />
#**TIME is the cumulative processor time used by the program on Linux<br />
#*<code>ps</code> supports traditional Unix options and GNU flags, which can be somewhat confusing<br />
#*Options for ps <br />
#**Display all processes running <code>ps -A/-e</code> both cause all processes on the system to be listed and output is identical<br />
#** Display processes belonging to a specific user: <code>ps -u user</code><br />
#**Display extra fields as a “full format” <code>ps -f</code> can be combined with other options <br />
#***<code>UID</code> User ID shows the username of the user running the command<br />
#***<code>PPID</code> parent process ID, the process ID of the process that spawned the process<br />
#***<code>C</code> processor utilization as % of usage over the lifetime of the process<br />
#***<code>STIME</code> start time of the process as per system time (only in 24h clock format if current day; otherwise uses MMDD format)<br />
#**Display long format <code>ps -l</code><br />
#***<code>F</code> Flags field<br />
#****<code>0</code> = no special flags for this process<br />
#****<code>1</code> = forked process, but did not call exec<br />
#****<code>4</code> = executable was executed with superuser privileges<br />
#***<code>S</code> State codes<br />
#****<code>D</code> Uninterruptible sleep (usually IO)<br />
#****<code>R</code> Running or runnable (on run queue)<br />
#****<code>S</code> interruptible Sleep (waiting for an event to complete)<br />
#****<code>T</code> Stopped, either by a job control signal or because it is being traced.<br />
#****<code>X</code> dead (should never be seen)<br />
#****<code>Z</code> Defunct (“zombie”) process, terminated but not reaped by its parent.<br />
#*** <code>UID</code> uses system number rather than human readable name<br />
#***<code>PRI</code> Priority is used by the kernel, rather than in userspace, to determine the next in queue for processing<br />
#***<code>NI</code> Niceness of a process describes how accommodating a process is to sharing time on the system.<br />
#**** This number ranges from +19 to -20, with 0 as the default.<br />
#**** In the scheduling system, niceness is used to determine how much processing to give to each process when it asks for it.<br />
#***<code>ADDR</code> the memory address of the process<br />
#***<code>SZ</code> the size of the image in pages<br />
#****how much virtual memory is in use by the process.<br />
#****Includes text, data, and stack space<br />
#****page size is a kernel variable and can be read by <code>getconf PAGE_SIZE</code><br />
#***<code>WCHAN</code> address of where a function is sleeping (only kernel functions)<br />
#***<code>RSS</code> amount of physical memory in use<br />
#*User-oriented format <code>ps -u</code> displays information in a more readable manner<br />
#**<code>%CPU</code> percent of processor time used by process divided by real time<br />
#**<code>%MEM</code> actual percentage of non-reserved physical RAM<br />
#**<code>VSIZE</code> virtual memory size used<br />
#*Virtual memory format <code>ps -v</code> displays information including number of:<br />
#**<code>MAJFL</code> major page faults of the process, indicating times it has used swap space<br />
#**<code>TRS</code> physical memory devoted to stack<br />
#**<code>DRS</code> physical memory devoted to non-executable code<br />
#* Recieve a process hierarchy <code>ps -AH</code> see parent-child relationships visually<br />
#*Choose your own adventure mode <code>ps -o uid,pid,ppid,command</code><br />
#**this command used to determine certain columns<br />
#*Piping ps can be piped as per normal to other programs:<br />
#**Ex; <code>ps -u darseife \| grep bash</code><br />
#**<code>pgrep</code> is a common utility that behaves similarly to the previous command<br />
#<code>top</code><br />
#*<code>top</code> is used to view processes as they execute in real time<br />
#**<code>-d</code> used to choose update frequency in seconds<br />
#**<code>-p</code> pid to monitor specifically<br />
#**<code>-n</code> run for N iterations<br />
#**<code>-r</code> command to renice a process within top<br />
#**<code>-s</code> to change the update rate within top<br />
#*<code>htop</code> can also be used as a next generation to top<br />
#<code>nice</code> and <code>renice</code><br />
#*The <code>nice</code> command is used to run a command with an adjusted niceness <br />
#**generally can’t give negative values to programs without superuser access<br />
#**<code>nice -n 12 ./script.sh</code> will add 12 to the default priority (0)<br />
#*<code>renice</code> is used to adjust niceness of running processes.<br />
#**<code>renice 0 -p 2772 3562</code> renices the processes 2772 and 3562 to 0<br />
#**<code>renice 1 -u darseife</code> renices the processes of user darsiefe to 1<br />
#Killing processes<br />
##<code>kill</code><br />
##*stopping a process currently running <code>-kill -s signal PID</code><br />
##** <code>PID</code> PID by default, kill sends a signal to a number of process IDs<br />
##**signal 1 is <code>SIGHUP</code> terminates interactive programs and causes daemons to reread config files<br />
##** signal 9 is <code>SIGKILL</code> stops all processes without performing a graceful exit or shutdown<br />
##**signal 15 is <code>SIGTERM</code> asks the process to terminate and to close out open files, run its shutdown script, etc.<br />
##*<code>kill</code> will only kill processes owned by the user, unless run with superuser access<br />
##<code>killall</code><br />
##*kill multiple things<br />
##*<code>killall vi</code> kills all running instances of vi for the user; when run as a superuser, this will kill any running instance of vi<br />
##* <code>killall -s 9 vi</code> kills all running instances of vi for the user by sending the <code>SIGKILL</code> signal to it; when run as a superuser, this will kill any running instance of vi.<br />
##*<code>killall -i vi</code> uses interactive mode to ask which processes to kill<br />
##*if you are running killall as root, try to use -i to prevent loss of other people’s work<br />
#Additional process control<br />
#*If you are running a program inside of a terminal, <code>[Ctrl + Z]</code> will pause and go to background<br />
#**To restart the process, run the command <code>fg</code> in the same terminal<br />
#** With multiple processes paused, <code>fg 1</code> or <code>fg 2</code> will bring back process 1 or 2<br />
#*Processes can be started in the background with &amp; at the end of the line; unlike paused processes, these will continue to run.<br />
<br />
<span id="linux-time-and-scheduling"></span><br />
=== Linux Time and Scheduling===<br />
<br />
#Overview<br />
#*Computer systems in general rely on a pair of clocks to operate<br />
#*Two clocks<br />
#**'''Hardware clock''' - initialized in BIOS/UEFI and kept updated via battery while computer is off<br />
#**'''Software clock''' - initialized at boot from BIOS/UEFI time, and is used to track time while running<br />
#* Typically, the clock of the computer is set using coordinated universal time (UTC)<br />
#**adjusted for use based on timezones and localization settings like DST before entering userspace<br />
#*Both the hardware clock and the software clock are not terribly reliable and tends to drift over time<br />
#*This can be a problem on x86_64 systems in general because many network services rely on accurate timekeeping<br />
#*This is made worse when having hundreds or thousands of servers requesting time from a single source could be bad<br />
#Unix Time Commands<br />
#*<code>date</code> outputs the current software clock time<br />
#*<code>date -u</code> outputs the UTC time<br />
#*<code>date 102715022019</code> sets the time to exactly Oct 27, 15:02, 2019 in local time<br />
#* <code>hwclock</code> used to retrieve the value of the hardware clock; must be done as root<br />
#*setting the hardware clock<br />
#** <code>hwclock --set--date 102715022019</code> is a manual set<br />
#**<code>hwclock -hctosys</code> set the hardware clock to the system clock<br />
#**<code>hwclock --synctohc</code> synchronizes system clock to hardware<br />
#*these are called by daemons and init on startup and shutdown<br />
#*both the system time and the hardware clock still suck and need a replacement<br />
#<code>NTP</code> Network Time Protocol<br />
#*NTP relies on a tree of servers (each level is called a stratum) from a hardware atomic clock server<br />
#*The further down the tree you are the less accurate the time is.<br />
#*One NTP server requesting time from a center will exchange packet via timestamp and the delay between communications is estimated<br />
#*It is best to pick NTP servers with a short delay<br />
#*Typically, a cluster of computer is requested from a local time server, which in turn is the only computer requesting times outside of the network<br />
#<code>timesyncd</code><br />
#*Traditionally, <code>ntpd</code> is the daemon that has provided client time synchronization<br />
#* Newer versions of Ubuntu use a service called <code>timesyncd</code><br />
#**uses a slightly different protocol than NTP, called <code>SNTP</code> (simplified network time protocol)<br />
#**configuration information is at <code>/etc/systemd/timesyncd.conf</code><br />
#**<code>timedatectl</code> is a way to view the current SNTP settings<br />
#*also more information related to this accessible with <code>sudo systemctl status systemd-timesyncd</code><br />
#Running jobs in the future: <code>cron</code><br />
#*<code>cron</code> like timesyncd or ntpd, cron is a daemon that runs continuously and looks for events to cause it to spring into action<br />
#*Most daemons are network services, but cron responds to temporal events<br />
#*A cron daemon wakes up once per minute, and examines its config files in:<br />
#**<code>/var/spool/cron/</code><br />
#**<code>/etc/cron.d/</code><br />
#**<code>/etc/crontab</code><br />
#*Executes command specified:<br />
#**System jobs - run as root, and perform system maintenance tasks (rotate logs, clean out <code>/tmp</code>, remove old files, update system)<br />
#**User jobs - run jobs in the future for users<br />
#*<code>/etc/crontab</code><br />
#** System cronjobs are controlled by this file<br />
#**Normally, begins with environment variable configuration, then several lines like:<br />
#***On a typical system <code>02 4 * * * root run-parts /etc/cron.daily</code><br />
#***On Ubuntu Desktop <code>25 6 * * * root test -x /usr/sbin/anacron || cd / &amp;&amp; run-parts --report /etc/cron.daily</code><br />
#**Fields for cron, in order<br />
#**#the minute of the hour<br />
#**#the hour of the day<br />
#**#day of the month<br />
#**#month of the year<br />
#**#day of the week<br />
#** different scripts for items are run from:<br />
#***<code>/etc/cron.daily</code><br />
#***<code>/etc/cron.weekly</code><br />
#***<code>/etc/cron.monthly</code><br />
#<code>anacron</code> a specific implementation of cron<br />
#*<code>anacron</code> is not intended to be a replacement for cron, it is a supplement<br />
#*cron works great on systems that are ''always on''<br />
#* <code>anacron</code> is smart in that it can tell if things are run<br />
#**if things are missed, it plays catchup<br />
#**light-night log rotation, for example, might never run<br />
#**anacron is intended to address this<br />
#***when run it checks to see when it last executed each program<br />
#***if a period greater than the programs interval has passed, the program gets run by anacron<br />
#** config file <code>/etc/anacrontab</code><br />
#***format: <code>1 5 cron.daily run-parts --report /etc/cron.daily</code><br />
#****first column: delay in minutes<br />
#****second column: frequency in days<br />
#<code>crontab</code> edit cronjobs for users<br />
#*cron jobs for the system are under <code>/etc/</code> but user ones are kept in <code>/var</code><br />
#*<code>crontab -e</code> edit your user cronfile<br />
#*root can edit other users with <code>crontab -u username -e</code><br />
#*<code>/etc/cron.allow</code> and <code>/etc/cron.deny</code> exist for whitelisting or blacklisting users from using <code>cron</code><br />
#**an empty <code>/etc/cron.deny</code> allows ALL USERS if present<br />
#**an empty <code>/etc/cron.allow</code> blocks ALL USERS if present<br />
#*if neither of these exist, then only root may use <code>cron</code><br />
#<code>at</code> a handy simple task timer<br />
#*cron and anacron can be overkill for simple tasks that need to be done once<br />
#*example:<br />
#**<code>at 5am Oct 31</code><br />
#**<code>at&gt; /home/darren/run-backup.sh</code><br />
#**<code>at&gt; echo &quot;job done&quot;</code><br />
#**<code>at&gt; mail -s &quot;backup job finished&quot; root</code><br />
#**<code>at&gt;</code> '''ctrl+d''' to finish<br />
#*<code>at now +1 hour</code> will run 1 hour from now relative to current system time<br />
#* other examples:<br />
#*#<code>at midnight</code><br />
#*# <code>at noon</code><br />
#*#<code>at teatime</code><br />
#* <code>aterm</code> to remove an upcoming command<br />
#* <code>atq</code> used to list pending commands<br />
<br />
<span id="log-files"></span><br />
===Log Files===<br />
<br />
#Intro<br />
#*Linux machines log files that record key details about system operation and are typically stored in <code>/var/log</code><br />
#*Logging on Linux system is generally performed by the syslog daemon, <code>syslogd</code> <br />
#**However, some daemons and programs perform logging independently of syslog<br />
#*Logging on Linux can be done locally to the computer, but in a networked environment, logs may be sent to a central logging server and by email<br />
#*Logging is a security action and is typically a part of the system worth hardening because it is exploitable by an attacker to hide their break-in<br />
#* Logs should be rotated, aka moved out of the system, to prevent <code>/var/log</code> from being filled completely<br />
#<code>syslog</code><br />
#*The basic idea of a system logger is to provide a unified means of a handling log files<br />
#*The daemon runs in the background and and accepts data delivered from servers and other programs<br />
#*Once the log daemon receives a message, it is classified and directed to an appropriate file<br />
#*<code>syslogd</code> might be managing several services at the same time and providing different rule-based logging based on priority of the messages received<br />
#*<code>syslogd</code> relies on the config file <code>/etc/syslog.conf</code><br />
#**Ubuntu uses <code>/etc/rsyslog.conf</code> for rocket-fast syslog<br />
#**the configuration file supports a ruleset, located at <code>/etc/rsyslog.d/</code> or <code>/etc/syslog.d/</code> respectively<br />
#**standard logging is done by daemon, with the format <code>facility.priority action</code><br />
#***example: <code>mail.* /var/log/mail.log</code><br />
#**facility parameter is the name of the type of program the daemon that generates messages to be logged <br />
#**#<code>auth/priv</code> security messages<br />
#**#<code>cron</code> from the cron daemon<br />
#**#<code>daemon</code> catch-all for various systems servers<br />
#**#<code>kern</code> kernel messages<br />
#**#<code>lpr</code> printer and CUPS messages<br />
#**#<code>mail</code> mail subsystems<br />
#**#<code>news</code> oldschool newsgroups<br />
#**#<code>syslog</code> internally generated messages<br />
#**#<code>uucp</code> unix-to-unix copy daemon<br />
#**#<code>local[0-7]</code> predefined facilities for custom configurations<br />
#** priority parameters include:<br />
#*** <code>deb</code> debugging messages<br />
#*** <code>info</code> normal operations<br />
#*** <code>notice</code> abnormalities that do not effect functionality<br />
#***<code>warn</code> tells that an error will occur is action is not taken<br />
#***<code>err</code> standard error for the daemon, non-urgent failure of the daemon causing an error message<br />
#***<code>crit</code> critical error causing a secondary system to fail urgently<br />
#***<code>alert</code> critical error causing a primary system to fail urgently<br />
#***<code>emerg</code> Panic message - system unusable<br />
#**the action parameter is a file or location on the network that will accept the message <br />
#***could be a file in <code>/var/log</code><br />
#***could also be a tty: <code>/dev/tty8</code><br />
#***could also be a computer on the network: <code>@logger.ndus.edu</code><br />
#*When a program sends a message to the system logger, it includes the priority: the message is only logged if it is of a certain level or higher<br />
#**Exceptions:<br />
#***<code>*</code> all codes<br />
#***= <code>(mail.=crit)</code> applies to this priority only<br />
#**You can also specify multiple facilities with the same priority by using a comma<br />
#**Multiple facilities separated by commas followed by a priority all follow the same priority<br />
#**A semicolon is used to specify a number of logging priorities <code>kern.info;kern.err</code> everything between info and error<br />
#<code>logger</code> - logging something else<br />
#*The <code>logger</code> command allows a user to log something else other than established facilities<br />
#*<code>logger shutting down for system maintenance</code><br />
#**will log to <code>/var/log/syslog</code> as <code>Nov 6 15:50:00 darseife-virtualbox darseife: shutting down for system maintenance</code><br />
#*<code>-i</code> records the PID of the logger process when run (useful when run from script)<br />
#*<code>-s</code> print the log message to standard error in addition to logging<br />
#* <code>-f /foo</code> logs to <code>/foo</code> instead of the system log<br />
#*<code>-p</code> specify both a facility and a priority for the system log message; <code>-p local3.info</code><br />
<br />
<span id="firewalling"></span><br />
===Firewalling===<br />
<br />
#TCP Wrappers<br />
#*TCP wrappers depend on two config files to work properly; these are <code>/etc/hosts.allow</code> and <code>/etc/hosts.deny</code><br />
#*the most common config for these is default <code>deny</code><br />
#**<code>ALL:ALL</code> in <code>/etc/hosts.deny</code><br />
#**<code>ALL:127.0 [::1]</code> in <code>/etc/hosts.allow</code><br />
#**specific services from a specific host enabled as the form <code>sshd:192.168.X.Y</code> in <code>/etc/hosts.allow</code><br />
#**all services from a domain as <code>ALL:.minotstateu.edu</code> in <code>/etc/hosts.allow</code><br />
#*Also used is specific allow<br />
#**nothing is listed in <code>/etc/hosts.allow</code><br />
#**specifically removed IP addresses in the form <code>ALL:165.234.X.Y</code> in <code>/etc/hosts.deny</code><br />
#* Secondary service-level blocking can be used to apply rules to services in the form <code>/etc/sshd.hosts</code><br />
#<code>iptables</code><br />
#*<code>iptables</code> is the kernel level netfilter firewall<br />
#*<code>iptables -L</code> lists the current ruleset<br />
#*<code>iptables -A</code> is used to APPEND new rules to a chain (takes a chain name)<br />
#*<code>iptables -I</code> inserts a rule (takes a chain name and a position)<br />
#**example <code>iptables -A INPUT -ptcp -dport ssh -j ACCEPT</code><br />
#*rule options<br />
#**<code>-p</code> the protocol used<br />
#**<code>-d</code> the destination port, can accept a range of port<br />
#**<code>-j</code> the action to take, includes ACCEPT, DENY, DROP<br />
#**<code>-s</code> specify a source address, can be either <code>192.168.0.1</code> or <code>192.168.0.0/24</code><br />
#**<code>-m</code> require the rule to match a specific property<br />
#***ex <code>iptables -m conntrack</code> allows filter rule to match based on connection state<br />
#**<code>--ctstate</code> define a list of connection states for the rule to match on<br />
#**#NEW - new connections<br />
#**#RELATED - new but related to a previous on<br />
#**#ESTABLISHED - a preexisting connection already in progress<br />
#**# INVALID - any malformed traffic or unidentified traffic<br />
#**<code>-j</code> tells iptable what to do with the packet given the rule<br />
#***ACCEPT - routes as normal as per IP header<br />
#***DROP - kills the packet without giving reply<br />
#***REJECT - sends an explicit error message<br />
# Uncomplicated Firewall (ufw)<br />
#*<code>ufw enable</code> to turn on ufw<br />
#*<code>ufw disable</code> to turn off ufw<br />
#*<code>ufw status</code> view current rules for ufw<br />
#*<code>ufw status verbose</code> reports logging level + access levels for each rule<br />
#* <code>ufw allow 80/tcp</code> sets a rule to allow tcp packets on port 80<br />
#*<code>ufw deny 22/udp</code> sets a rule to reject udp packets on port 22<br />
#*<code>ufw deny ssh</code> sets a rule to reject all ssh protocol packets<br />
#*<code>ufw allow 6000-6007/tcp</code> sets a rule with a range to allow tcp packets from 6000-6007<br />
#*<code>ufw allow from 15.15.15.1 to any</code> sets a rule allowing all traffic from 15.15.15.1<br />
#*<code>ufw status numbered</code> returns a numbered list of rules in the set<br />
#*<code>ufw delete 2</code> removes rule number 2 in a list<br />
#*<code>ufw logging {on/off}</code> turns the logging on or off; usually caught by syslogd/rsyslog; logging is set to <code>low</code> by default<br />
#* <code>ufw logging {low/medium/high/full</code> sets logging verbosity of ufw;<br />
#**note that logging is rate limited at <code>medium</code> and not rate limited at <code>high</code><br />
<br />
<span id="bootinitialization"></span><br />
===Boot/Initialization===<br />
<br />
# Slackware - SysV Init<br />
#*System 5 style rc script via <code>init</code><br />
#*<code>PID 1</code> = <code>init</code> which is the process that starts all other processes when the system boots<br />
#* <code>init</code> is responsible for processing <code>/etc/inittab</code> to see how to start the system and at what runlevel<br />
#*Things <code>init</code> does<br />
#**Runs the script <code>/etc/rc.d/rc.s</code> to determine to prepare the system before loading to the desired runlevel<br />
#**Enables virtual memory for the system and mounts filesystems in <code>/etc/fstab</code><br />
#**Loads kernel modules<br />
#**Near the end of the script <code>rc.s</code>, init will run scripts related to the runlevel the system is entering -for example, <code>rc3.d</code> is run at runlevel 3<br />
#SystemD<br />
#*Ubuntu, CentOS, RedHat, and others have moved from SysV init process to the SystemD initialization process<br />
#*Rather than running many shell scripts during init, the systemD process uses one monolithic program to control startup of services<br />
#*The way the services are started are controlled by a configuration file<br />
#*systemd uses ''units'' and ''targets'' rather than scripts and runlevels <br />
#**there are 11 different types of units:<br />
#***<code>mount</code> unit defines a mountpoint on the system named with the mount path with slashes replaced as dashes<br />
#***<code>swapspace</code><br />
#***<code>device</code> a device in the linux device tree<br />
#***<code>socket</code> fifo pipe to an internet port with an associated service<br />
#*** <code>service</code> a service or application on the system<br />
#**targets are used to stop and start the service and group units together<br />
#**systemd identifies units by name and type<br />
#***example ssh could have an ssh.service and an ssh.socket<br />
#*<code>systemctl</code> - can list current units loaded on the system<br />
#*config files - <code>/lib/systemd/system</code><br />
#*modify how a unit functions - <code>/etc/systemd/system</code></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=VASP_and_VMD_on_CCAST&diff=130VASP and VMD on CCAST2022-10-21T17:48:48Z<p>Sysadmin: </p>
<hr />
<div>__TOC__<br />
<br />
{|<br />
|-<br />
|<br />
'''Misc. Tools'''<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| cd foldername<br />
| lets you go into a folder<br />
|-<br />
| cd ..<br />
| Backs you out of folder you are in<br />
|-<br />
| cd -<br />
| Goes back to previous directory<br />
|-<br />
| cd ~<br />
| Goes to home directory<br />
|-<br />
| freenodes<br />
| in ccast<br />
|-<br />
| vi filename<br />
| Lets you go into file to edit<br />
|-<br />
| i<br />
| When in “vi” this initiates the editing<br />
|-<br />
| :wq<br />
| This lets you stop editing<br />
|-<br />
| “Ctrl+c” press on keyboard, not a typing thing<br />
| Lets you stop any current running process<br />
|-<br />
| pwd<br />
| Gives you current directory location<br />
|-<br />
| wc<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| ls<br />
| Lists what is in the current directory<br />
|-<br />
| ls -lt<br />
| Lists what is in the current directory sorting from newest to oldest<br />
|-<br />
| ls –lt W*<br />
| Lists everything that starts with a W sorting from newest to oldest. (Can be used for other letters as well)<br />
|-<br />
| ls -lisanh<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisanh filename<br />
| gives information on the specific file<br />
|-<br />
| ls -lisan<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisan filename<br />
| gives information on the specific file<br />
|-<br />
| ls -a<br />
| shows all files, including hidden ones<br />
|-<br />
| babel –ixyz file.xyz –opdb file.pdb xyz to pdb<br />
|<br />
Using babel, converts the input file of .xyz to outputfile of .pdb<br />
<br />
That is where the –ixyz and –opdb comes from<br />
|-<br />
| “shift+alt+left click”<br />
| For slecting just comlumn or just row information you want exactly<br />
|-<br />
| egrep '163|166' OS_STRENGTH<br />
| grep technique<br />
|-<br />
| more filename<br />
| Lets you see what is in the file<br />
|-<br />
| more ~/bin/filename<br />
| Lets you see the file in a location not in your current directory.<br />
|-<br />
| more filename | grep ### or value | grep again if desired<br />
| shows file and grabs lines with same input you put after the grep command<br />
|-<br />
| cp /Location/filename .<br />
| Copies a file from a different directory location and input it into the current directory. Don’t forget the PERIOD!<br />
|-<br />
| P /Location/filename ./newfilename<br />
| Copies a file from a different directory location and input it into the current directory and rename it in one step<br />
|-<br />
| scp sourceUser:/sourceLocation/filename .<br />
| Takes file from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/* .<br />
| Takes ALL files from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/*.filetype .<br />
| Takes ALL files with that specific file type from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp * destinationUser:/destinationLocation<br />
| Sends all files from source directory to destination directory. Must be in source directory when typing this line<br />
|-<br />
| scp * username@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/username/DIMER/DIMabsorptionspec<br />
| Example of scp<br />
|-<br />
| sbatch corifilename<br />
| Lets you submit a file in cori using a corifile made by others<br />
|-<br />
| grep specificdetail filename<br />
| Grabs the specific detail from the file you chose<br />
|-<br />
| sort -n -r filename<br />
| sorts the file in ascending order<br />
|-<br />
| babel BondLength.xyz -O out.xyz --separate<br />
| THIS IS CORRECT WAY THE SEP WORD IS SPELLED INCORRECTLY BUT IS CORRECT FOR BABEL<br />
|-<br />
| awk '{ if($4 &gt;= 4.0000 &amp;&amp; $4&lt;=5.0000) print }' OS_STRENGTH1<br />
| lets you get lines of data that contain the range from 4-5 from column 4<br />
|-<br />
| users<br />
| lets you see what users are on<br />
|-<br />
| clear<br />
| lets you clear screen<br />
|-<br />
| who<br />
| who is on and more detail about them<br />
|-<br />
| w<br />
| all the people and differing details about them<br />
|-<br />
| id<br />
| lets me see what groups I am in<br />
|-<br />
| freenodes<br />
| lets you see if nodes are available or not<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6335cce9234bc4ef.png|439x225px]]<br />
|<br />
|-<br />
| pbsnodes –avSj | sort -n<br />
| shows available nodes, the | sorts it into order<br />
|-<br />
| pbsnodes –avSj | sort –n | more<br />
| does this in page format rather than all at once<br />
|-<br />
| module avail<br />
| what software is available<br />
|-<br />
| $HOME<br />
| lets you go to your login directory “home” directory<br />
|-<br />
|<br />
thunder.ccast.ndsu.edu<br />
<br />
meade.erickson<br />
| to get into putty<br />
|-<br />
| C:\Users\meade.erickson\Documents\OriginLab\User Files\<br />
| Origin themes location<br />
|-<br />
| '''Explanations'''<br />
| <br /><br />
|-<br />
| fermi<br />
| fermi energy is referring to the energy difference between the highest and lowest occupied single-particle states. ([https://en.wikipedia.org/wiki/Fermi_energy <u>https://en.wikipedia.org/wiki/Fermi_energy</u>])<br />
|-<br />
| '''Explanations Input Files:'''<br />
| <br /><br />
|-<br />
| INCAR<br />
| It is the central input file of VASP. It determines &quot;what to do and how to do it&quot;. ([https://www.vasp.at/wiki/index.php/INCAR <u>https://www.vasp.at/wiki/index.php/INCAR</u>])<br />
|-<br />
| POSCAR<br />
| <br /><br />
|-<br />
| POTCAR<br />
| <br /><br />
|-<br />
| KPOINTS<br />
| <br /><br />
|-<br />
| '''Explanations Output Files:'''<br />
| <br /><br />
|-<br />
| CONTCAR<br />
|<br />
Contains actual coordinates, velocities, and predictor corrector coordinates. 1<sup>st</sup> block is lattice parameters and atom coordinates. 2<sup>nd</sup> block Initial Velocites for atoms. 3<sup>rd</sup> block Predictor-corrector coordinates.<br />
<br />
Written at end of each job. Used as the POSCAR file for next MD job.<br />
|-<br />
| WAVECAR<br />
| Provides wavefunctions. we usually use it for spectrum data ([https://www.vasp.at/wiki/index.php/WAVECAR <u>https://www.vasp.at/wiki/index.php/WAVECAR</u>])<br />
|-<br />
| OS_STRENGTH<br />
| Information about the OS_STRENGTH<br />
|-<br />
|<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8bf47c7304419724.png|354x154px]]<br />
|<br />
From Wikipedia<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
In spectroscopy, '''oscillator strength''' is a dimensionless quantity that expresses the probability of [https://en.wikipedia.org/wiki/Absorption_(electromagnetic_radiation) <u>absorption</u>] or [https://en.wikipedia.org/wiki/Emission_spectrum <u>emission</u>] of [https://en.wikipedia.org/wiki/Electromagnetic_radiation <u>electromagnetic radiation</u>] in transitions between [https://en.wikipedia.org/wiki/Energy_level <u>energy levels</u>] of an atom or molecule<sup>[</sup>[https://en.wikipedia.org/wiki/Wikipedia:Accuracy_dispute#Disputed_statement <sup>''<u>dubious</u>''</sup>]<sup>'' – ''</sup>[https://en.wikipedia.org/wiki/Talk:Oscillator_strength#Wrong_definition_oscillator_strength <sup>''<u>discuss</u>''</sup>]<sup>]</sup>.<sup><u>[https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Demtr%C3%B6der2003-1 [1]][https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Robinson1996-2 [2]]</u></sup> The oscillator strength can be thought of as the ratio between the quantum mechanical transition rate and the classical absorption/emission rate of a single electron oscillator with the same frequency as the transition.<br />
|-<br />
| POTIM (in INCAR file)<br />
| timestep in femtoseconds. So POTIM=1 is 1 femtosecod<br />
|-<br />
| '''Density of States'''<br />
| <br /><br />
|-<br />
| To get DOS you need to run the correct calculations in VASP using 4 files. INCAR, POSCAR, POTCAR, OUTCAR<br />
| <br /><br />
|-<br />
| Once you get output from your submitted information to cori or photon you can follow these steps to get Density of states<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| Grabs the number of bands (we need this for later) (STATES)<br />
|-<br />
| vi input_overlap<br />
| editing input_overlap<br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| Number of Bands from the grep NBA OUTCAR line<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| quitting editing input_overlap<br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| getting energy pop from different location<br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| wc STATES<br />
| <br /><br />
|-<br />
| grep NELE OUTCAR<br />
| record number of electrons<br />
|-<br />
| grep NIO OUTCAR<br />
| record number of ions<br />
|-<br />
| grep E-fer OUTCAR<br />
| record the three values and their names. If this line doesn’t work the try next line)<br />
|-<br />
| grep ferm OUTCAR | tail –n 1<br />
| record the three values and their names<br />
|-<br />
| ~/bin/DRESS_DOS_NORM<br />
| Executes the DOS code from different file location<br />
|-<br />
|<br />
Number of states from grep NBA OUTCAR<br />
| grep NBA OUTCAR line<br />
|-<br />
| -20<br />
| <br /><br />
|-<br />
| 20<br />
| <br /><br />
|-<br />
|<br />
E-fermi info from grep E_fermi<br />
<br />
<br /><br />
|<br />
To literally line up DOS and STATES, you may need to type in 0 for both fermi and alpha+beta.<br />
<br />
<br />A more standard way is to use the middle point energy between HOMO and LUMO i.e. (E_HOMO+E_LUMO)/2 for fermi and 0 for alpha+beta. In this way, the mid point of DOS is at zero, anything below zero is occupied and above zero is unoccupied.<br />
|-<br />
|<br />
alpha+beta info from grep E_fermi<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| .01<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
|<br />
Number of ions from grep NIO OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
Number of electrons from grep NELE OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| 3<br />
| <br /><br />
|-<br />
|<br />
gnuplot ~/bin/gnuprog_DOS.fermi<br />
<br />
<br /><br />
| Plotting the DOS (This is done in NERSC or PHOTON)<br />
|-<br />
|<br />
ps2pdfwr DOS.ps<br />
<br />
<br /><br />
| Converting DOS file from .ps to .pdf<br />
|-<br />
|<br />set out MEADESDOS.ps MAY NOT NEED THIS NOT SURE YET<br />
set style line 4 lw 4 lc -1; unset key; plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
set style line 4 lw 4 lc -1<br />
<br />
unset key<br />
<br />
plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4;<br />
<br />
set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0<br />
<br />
set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0<br />
<br />
set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0<br />
<br />
set tics font &quot;,30&quot;<br />
<br />
set ytics offset 0,0,0<br />
<br />
set xtics offset -0.4,-1,0<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.518567):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.518567):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.173350):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.17335):4 w l title &quot;Wet Monomer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Wet Monomer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
|<br />
I am initially specifying a style that I will be using. In this case chose style 4 and am editing it to what I want.<br />
<br />
lw stands for linwidth, lc line color, not sure where the library is but lc -1 is black, 0=red I think and find out other through trial and error<br />
<br />
lw is just multiplyer, so lw 4 is 4 times larger than default.<br />
<br />
unset key gets rid of key,<br />
<br />
[#,#] is the range of my x axis graph.<br />
<br />
($#-###): $# is the column I want to take data from which is being subtracted by the amount of shift I am giving to the data (we want data to be centered at 0 sometimes and so for DOS we take the middle between the start of non occupied plus the end of occupied and divide by 2 to get our shift. (Notoccupied + occupied)/2 ) and the number after the “:” is the column I am correlating the X and Y axis with. So when we plot stuff we need to specify what is the x and what is the y axis.<br />
<br />
not sure exactly what the “w i” portion does but it helps us change from points to lines and then is what fills in the red portion of our graph. (The “i” stands for impulses. Add an extra “lw 2’ to increase the line width as shown in green text.) “w l” does the same thing but DOESN”T fill out the the electron density portion.<br />
<br />
xlabel and y labels are the axis titles and offsets help position them due to the changing sizes of the axes and tics.<br />
<br />
Tics are the numbers on the Y and x axis so ytics offsets them and xtics does the horizontal axis change.<br />
<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
| '''Absorption Spectrum - Transition Energy eV – Wavelength nm'''<br />
| <br /><br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
|<br />
copying INCAR-gm into the directory as long as you are in it<br />
<br />
MAKE SURE<br />
<br />
LWAVE= .TRUE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br />
username@cori07:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec/TEST<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Or make sure the INCAR has this Ag+Carboxyl<br />
<br />
#ISIF=7<br />
<br />
#PREC = HIGH<br />
<br />
ENCUT= 300<br />
<br />
IBRION =3<br />
<br />
POTIM=0.2<br />
<br />
NSW = 30000<br />
<br />
ALGO = NORMAL #48<br />
<br />
ISMEAR = 0<br />
<br />
SIGMA = 0.01<br />
<br />
LREAL = A<br />
<br />
#ISTART = 0<br />
<br />
LWAVE = .TRUE.<br />
<br />
LCHARG = .FALSE.<br />
<br />
EDIFF=1E-6<br />
<br />
EDIFFG=-0.05<br />
<br />
LVTOT = .FALSE.<br />
<br />
#LVDW=.TRUE.<br />
<br />
#SPRING = -5<br />
<br />
#LCLIMB = .TRUE.<br />
<br />
#IMAGES = 7<br />
<br />
#IOPT=7<br />
<br />
#ICHAIN=0<br />
<br />
NELM=40<br />
<br />
#ISPIN=2<br />
<br />
ISYM=0<br />
<br />
NPAR=4<br />
<br />
<br /><br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|345x215px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| It is necessary to have the ‘spectrum’ file made which comes from a healthy WAVECAR which is used to generate OS_STRENGTH. After OS_STRENGHT we make the spectrum ~/bin/SPECTRUM_imp3 file.<br />
| <br /><br />
|-<br />
| Assuming you have conducted a calculation which generated a healthy WAVECAR file.<br />
| <br /><br />
|-<br />
| If healthy WAVECAR exists then…<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| record Number of bands<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
|<br />
1<br />
<br />
Number of Bands<br />
<br />
1<br />
| 36740210<br />
|-<br />
|<br />
‘esc’ key<br />
<br />
:wq<br />
| <br /><br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| <br /><br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| ./osc_str.exe<br />
| This makes the OS_STRENGTH (oscillator strength)<br />
|-<br />
| grep '2.0000 0.0000' OS_STRENGTH &gt; OS_STRENGTHvc<br />
|<br />
Takes the orbital pairs from 2.0000 to 0.0000 into new file.<br />
<br />
<br /><br />
|-<br />
| cp OS_STRENGTH OS_STRENGTH1<br />
| making sure I have original OS_STRENGTH<br />
|-<br />
| cp OS_STRENGTHvc OS_STRENGTH<br />
| making sure I have file OS_STRENGvc kept for 2.0000 – 0.0000 but it will be our OS_STRENGTH<br />
|-<br />
| wc OS_STRENGTH<br />
| Record First number<br />
|-<br />
| ~/bin/SPECTRUM_imp3<br />
| <br /><br />
|-<br />
|<br />
Number from OS_STERNEGTH<br />
<br />
.1<br />
<br />
10<br />
<br />
.001<br />
<br />
HOMO<br />
|<br />
Should be able to find HOMO from STATES file (or energy_pop)<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Can change the “.001” to “0.1 or 0.01” to smooth out the line<br />
|-<br />
| a ‘spectrum’ file should have been made. This holds all the spectrum data. The spectrum can be visualized using these commands in Cori. If you want to visualize in photon then use the gnuplot information below<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE<br />
|<br />
IN CORI YOU CAN USE THESE<br />
<br />
THEN SCP them to photon, then the desktop!<br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE_nm_200_800<br />
|<br />
|-<br />
| ps2pdfwr SPE.ps<br />
|<br />
|-<br />
| ps2pdfwr SPE_nm.ps<br />
|<br />
|-<br />
| scp * username@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec<br />
| example make sure you are in the directory of the files you want to send. Send it to a premade directory in photon<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
|<br />
FOR PHOTON USE THESE STEPSWe are plotting the information of spectrum to get the spectrum in wavelength of nanometers nm and transition energy in electron volts eV<br />
| <br /><br />
|-<br />
| more spectrum<br />
|<br />
(Be sure to be in correct directory)<br />
<br />
Column 1 is the spectrum in wavelength of nm<br />
<br />
column 2 is the spectrum in transition energy of eV<br />
<br />
column 3 is absorbance/absorption in a.u. (absoption units). Absorbance goes with wavelength, absorption goes with transition energy<br />
<br />
column 4 idk what this is.<br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| plot ‘spectrum’ 2:3<br />
| This gives you transition energy in x axis and absorption in a.u for yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [0:10] 'spectrum' using 2:3 w l title &quot;Transition Energy&quot; ls 4; set xlabel 'Transition Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorption a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| This is transition energy in x axis and absorption in yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [200:800] 'spectrum' using 1:3 w l title &quot;Absorption Spectrum&quot; ls 4; set xlabel 'Wavelength, nm' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorbance, a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| nm in x axis and absorbance in yaxis<br />
|-<br />
| '''UV-Vis Absorption Spectrum'''<br />
| <br /><br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
We did one calculation already but we need to do a “geometry optimization” for the next one.<br />
<br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
| copying INCAR-gm into the directory as long as you are in it<br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|352x219px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| '''Generating CANDIDATES file for Photofragmentation'''<br />
| <br /><br />
|-<br />
| grep –A 700 fermi OUTCAR | tail –n 700 &gt; STATES1<br />
| Fermi information from OUTCAR into a file called STATES 1. Not sure what the –A or 700 mean. But the tail is choosing a specific section of the data, the “tail” end of the chosen information from the OUTCAR file.<br />
|-<br />
| more STATES1<br />
| Determine HOMO and LUMO, HOMO is the highest orbital with a 2, LUMO is the first 0. In my case HOMO is 112 and LUMO is 113<br />
|-<br />
| NOT SURE HOW HE GENERATED STATES FILE<br />
| <br /><br />
|-<br />
| head STATES<br />
| Record the lowest number, in my case it is 81 for DIMER<br />
|-<br />
| tail STATES<br />
| Record the Highest number, in my case it is 160 for DIMER<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| 160<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| more ~/bin/ADVICE.spectrum<br />
| This will make spectrum. This will show 3 lines of code. Literally copy one line and press enter starting with the “module swap PrgEnv-intel PrgEnv-gnu”<br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| copying templates and inputting them into the directory you are in<br />
|-<br />
| ./osc_str_CHEM676.exe<br />
| This reads binary WAVECAR file, Reads input_overlap, creates Oscillator strength<br />
|-<br />
| ls –lt | head<br />
| This should show a fresh OS_STRENGTH was made<br />
|-<br />
| grep ‘2.0000 0.0000’ OS_STRENGTH &gt; OS_NEW<br />
| In the OS_STRENGTH file we will see lines of no transition from occupied to unnoccupied. What we want to see is the lines that have an occupied originally (2.0000) and then are unoccupied (0.0000). This shows excitation. The lines that meet this requirement are then inputted into OS_NEW<br />
|-<br />
| head OS_NEW<br />
| This should show a range of orbitals that have been occupied and are now unoccupied. With the differing orbitals involved. Columns 1 and 2 are the initial and final orbitals that are involved with the “hopping” of electrons. The 3rd column is oscillator Strength. 4th Column is Transition energy.<br />
|-<br />
| mv OS_NEW OS_STRENGTH<br />
| We don’t need all OS_STRENGTH data, just the stuff we “grepped” in OS_NEW. This overrides OS_STRENGTH with information from OS_NEW.<br />
|-<br />
| wc OS_STRENGTH<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| head –n 30 OS_STRENGTH<br />
| In this particular case we only want the top 30 rather than the original value the first column gave was. In my case it was about 1536.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4<br />
| To sort in ascending order of transition energy. 4 is the 4<sup>th</sup> column. I don’t know what –gk means.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4 &gt; CANDIDATES<br />
| Inputs this information into CANDIDATES file<br />
|-<br />
| '''Orbital Visualization'''<br />
| <br /><br />
|-<br />
| I don’t have all information available here, for example I personally didn’t make the PARCHG files.<br />
| <br /><br />
|-<br />
|<br />
Guidance to make PARCHG files:<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; mkdir ORBITALS<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp WAVECAR ORBITALS/<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp CONTCAR ORBITALS/POSCAR<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp POTCAR ORBITALS/<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp *.sh ORBITALS/<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp KPOINTS ORBITALS/<br />
<br />
&gt; cd ORBITALS/<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; grep NBA ../OUTCAR<br />
<br />
k-points NKPTS = 1 k-points in BZ NKDIM = 1 number of bands NBANDS= 160<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; cp ~/bin/INCAR/INCAR-pc ./INCAR<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; tail INCAR<br />
<br />
IBRION=2 #conjugate-gradient algorithm used to relax ions (bad<br />
<br />
NSW=0 #number of ionic steps<br />
<br />
POTIM= .2 #time step in fs<br />
<br />
EDIFFG=-0.001 #minimum energy difference between ionic iterations<br />
<br />
EINT= -4.4 -0.5<br />
<br />
EINT= -6 2 <br />
<br />
LVTOT = .TRUE.<br />
<br />
username@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; sbatch cori_debug.sh <br />
<br />
Submitted batch job 32413471<br />
<br />
many thanks for the new version!<br />
<br />
The main question in your email is orbitals with VASP:<br />
<br />
a. generate or use premade healthy WAVECAR <br />
<br />
(it is an analog of .chk in gaussian)<br />
<br />
b. in a new directory, use template of the INCAR for orbitals and run vasp once again.<br />
<br />
the templates are named ~/bin/INCAR/INCAR_pc and similar names.<br />
<br />
(it is an analog of the cubegen procedure in gaussian)<br />
<br />
c. upon successful run, there appear files nammed PARCHG.XXXX.ALLK, where XXXX stands for the index of orbital, e.g. PARCHG..01112.ALLK<br />
<br />
(those files are analogs of the .cube files in gaussian. note that vasp DOES not generate literal .cube files!)<br />
<br />
d. download these files and open then in the VMD - the same software you have used for the trajectories.<br />
|<br />
|-<br />
| Step 2. Changing PARCHG Files to cube files (For Orbital Visualization)<br />
| From what I have seen we need cube files to show in guassview AND VMD<br />
|-<br />
| Once obtained PARCH.####.ALLK Files then use following command in PHOTON to convert the orbitals you care about to cube!<br />
| <br /><br />
|-<br />
| ~/vtstools/chg2cube.pl filename<br />
| <br /><br />
|-<br />
| 3<br />
| selecting PARCHG<br />
|-<br />
| 1 6 7 8<br />
| (These are the atomic numbers for the atoms in your system. In my case I have Hydrogen, Nitrogen, Oxygen, and Carbon<br />
|-<br />
| Your cube file is then generated<br />
| <br /><br />
|-<br />
| SEND cube files, POSCAR (OR CONTCAR), CHGCAR FILES TO DESKTOP<br />
| <br /><br />
|-<br />
| Step 3. Using VMD to generate orbitals<br />
| <br /><br />
|-<br />
| Open VMD<br />
| <br /><br />
|-<br />
|<br />
In the VMD Main window go to “File” “new Molecule” Upload your<br />
<br />
1.POSCAR (OR POSSIBLY CONTCAR): By selecting VASP_POSCAR under the “Determine file type:”<br />
<br />
2. CHGCAR: By selecting VASP_CHGCAR under the “Determine file type:”<br />
<br />
3. USE THIS ONE PARCHG: By selecting VASP_PARCHG under the “Determine file type:” (Even though the file ends in .cube we are classifying it as PARCHG)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b0283f3c67e04e7d.png|219x134px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_eae9e77b87b8627b.png|222x103px]]<br />
|-<br />
|<br />
After all three are uploaded,<br />
<br />
in the VMD Main window go to “Graphics” and then “Representations”<br />
| <br /><br />
|-<br />
|<br />
In the Graphical Representations window follow these three examples.<br />
<br />
<br />(Change the file you are using by the dropdown menu of “selected Molecule”)<br />
|<br />
The red lettering means that PARCHG Isosurface isn’t being shown. You can double click on the red lettering to make it black and showing on the OpenGL Display Window. Double click again to revert to black.<br />
<br />
Parchrg can be shown and can be visualized in conjungtion with the CHGCAR file<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_e7a558f4dc169c4e.png|121x260px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b15ffb6b911b765.png|124x266px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3734231f026ab64e.png|124x265px]]<br />
|<br />
|-<br />
|<br />
-You can change the iso value in CHGCAR to get varying orbital visualizations.<br />
<br />
-Play with it.<br />
<br />
-Change the color by changing the ColorID to different ones.<br />
<br />
-(Range in CHGCAR doesn’t matter)<br />
|<br />
For Monomer Project Show<br />
<br />
all index 52 47 29 44 38 28 18 56 64<br />
<br />
H52<br />
<br />
H47<br />
<br />
H29<br />
<br />
H44<br />
<br />
H38<br />
<br />
H28<br />
<br />
H18<br />
<br />
H56<br />
<br />
H64<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c5f67d67a9a89fa7.png|355x180px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d4c6c68549d8e025.png|357x209px]]<br />
| <br /><br />
|-<br />
| To get rid of opaque water molecuels make both of these in “Graphics” “Materials” 123 is for faded bods and 124 is for the orbitals<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8d905765b5183c3.png|142x229px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_449e5be8bb4a3541.png|139x222px]]<br />
|<br />
|-<br />
| For my Monomer_H2O first frame hide these atoms if you don’t want in view<br />
| <br /><br />
|-<br />
|<br />
O108<br />
<br />
O99<br />
<br />
O94<br />
<br />
O101<br />
<br />
O81<br />
<br />
O91<br />
<br />
O90<br />
<br />
O86<br />
<br />
O82<br />
<br />
O87<br />
<br />
O79<br />
<br />
<br />H73<br />
<br />
H45<br />
<br />
H62<br />
<br />
H51<br />
<br />
H58<br />
<br />
H50<br />
<br />
H48<br />
<br />
H14<br />
<br />
H22<br />
<br />
H36<br />
<br />
H42<br />
<br />
H31<br />
<br />
H20<br />
<br />
H23<br />
<br />
H35<br />
<br />
H30<br />
<br />
H54<br />
<br />
H59<br />
<br />
H25<br />
|<br />
all not index ### ### ### ###...<br />
<br />
all not index 108 99 94 101 81 91 90 86 82 87 79 73 45 62 51 58 50 48 14 22 36 42 31 20 23 35 30 54 59 25<br />
<br />
<br />backbone <br />
<br />
index 97 103 11 9 10 13 12 96 1 84 78 3 4 100 77 0 8 2 5 6 7<br />
|-<br />
| '''These are for the figure first frame for still seeing the water atoms but hazy'''<br />
| <br /><br />
|-<br />
|<br />
H22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46<br />
<br />
<br />O90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
| 22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46 90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
|-<br />
| '''Index for Diagram for pics Middle Frame for H20 molecules'''<br />
| <br /><br />
|-<br />
|<br />
H63<br />
<br />
H33<br />
<br />
H17<br />
<br />
H21<br />
<br />
60<br />
<br />
68<br />
<br />
42<br />
<br />
38<br />
<br />
34<br />
<br />
50<br />
<br />
72<br />
<br />
73<br />
<br />
16<br />
<br />
61<br />
<br />
55<br />
<br />
71<br />
<br />
59<br />
<br />
69<br />
<br />
40<br />
<br />
14<br />
<br />
39<br />
<br />
74<br />
<br />
67<br />
<br />
53<br />
<br />
49<br />
<br />
41<br />
<br />
52<br />
<br />
27<br />
<br />
15<br />
<br />
31<br />
<br />
23<br />
<br />
54<br />
<br />
62<br />
<br />
32<br />
<br />
64<br />
<br />
30<br />
<br />
26<br />
<br />
57<br />
<br />
37<br />
<br />
51<br />
<br />
58<br />
<br />
70<br />
<br />
43<br />
<br />
25<br />
<br />
35<br />
<br />
46<br />
<br />
45<br />
|<br />
O80<br />
<br />
87<br />
<br />
99<br />
<br />
81<br />
<br />
88<br />
<br />
105<br />
<br />
104<br />
<br />
91<br />
<br />
90<br />
<br />
106<br />
<br />
97<br />
<br />
108<br />
<br />
102<br />
<br />
89<br />
<br />
93<br />
<br />
101<br />
<br />
92<br />
<br />
107<br />
<br />
78<br />
<br />
109<br />
<br />
86<br />
<br />
83<br />
<br />
94<br />
<br />
63 33 17 21 60 68 42 38 34 50 72 73 16 61 55 71 59 69 40 14 39 74 67 53 49 41 52 27 15 31 23 54 62 32 64 30 26 57 37 51 58 70 43 25 35 46 45 80 87 99 81 88 105 104 91 90 106 97 108 102 89 93 101 92 107 78 109 86 83 94<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_783e625ce0b8a321.png|182x247px]]<br />
|-<br />
| '''Preparation of TDESMD Orbital Files and Job Submission'''<br />
| <br /><br />
|-<br />
| mkdir TDESMD_111_114<br />
| We will now use the information of the CANDIDATES file and create new folder for each scenario. You create a file titled using the orbitals involved. In my case the first scenario is with orbitals 111 and 114<br />
|-<br />
| cd TDESMD_111_114<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/INCAR_EXCITED<br />
| Taking INCAR_EXCITED file from location. In my case it is here.<br />
|-<br />
| vi INCAR_EXCITED<br />
| Edit the INCAR_EXCITED file to make it look like<br />
|-<br />
| i<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_79a32df6f9f33bb0.png|303x201px]]<br /><br />
|-<br />
| <br /><br />
|<br />
Change FERWE=#### to match the total number of orbitals/bands. In this example we have a total number of 160 bands/orbitals.<br />
<br />
<br />
The first segment of FERWE, “110*1” says we have the first 110 orbitals occupied. The second portion is orbital number 111 which we have listed as a “0” signifying this orbital is unnoccopied for our job submission. Then orbital number 112 is occupied with “1”. Then orbital number 113 is unoccupied with “0”. Then orbital number 114 is occupied with “1”. The last part of “46*0” is talking about the other orbitals not mentioned. These orbital should add up to 160. These numbers change with the differing orbitals designating in the CANDIDATES FILE.<br />
<br />
The main thing that is changing is the position of one occupied orbital and one unoccupied orbital when the compound is in ground state. In this example number 111 is usually occupied in ground state and we are moving the electron to 114 which is unoccupied in ground state.<br />
|-<br />
| <br /><br />
| ONE more example is as follows. With TDESMD_102_115<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_51e84877e4374ca6.png|300x199px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POSCAR ./CONTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/KPOINTS<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/debug1.sh<br />
| <br /><br />
|-<br />
| sbatch debug1.sh<br />
| <br /><br />
|-<br />
| '''Creating Videos from movieAll.xyz files and uploading them on youtube'''<br />
| <br /><br />
|-<br />
| scp –r TDESMD_###_### DestinationUser:/DestinationLocation<br />
|<br />
Starting in cori<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Using this to send files from cori to photon. We specifically want the movieALL.xyz files.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
This is sending the WHOLE directory and everything it contains<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| mkdir TDESMD_###_###<br />
|<br />
In DESKTOP<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
generate TDESMD_###_### directories<br />
|-<br />
| scp username@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_##_###/movieALL.xyz .<br />
| WHILE IN THE TDESMD DIRECTORY THAT YOU WANT TO BE THE DESTINATION take “movieALL.xyz” from photon and put into specific directory<br />
|-<br />
| <br /><br />
| Once the file is in your desktop you can go to your windows folders and open it to check if it is there.<br />
|-<br />
| In windows search bar<br />
| <br /><br />
|-<br />
| VMD<br />
|<br />
type vmd in windows search bar and open it.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
VMD will open with multiple screens.<br />
<br />
<br /><br />
|-<br />
| We will focus on this screen first to open all the extra screens we need. (VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_5f31d31c2c72c623.png|244x114px]]<br />
|-<br />
| Press “File”, “New Molecule,”<br />
| <br /><br />
|-<br />
| On the window that comes up (Molecule File Browser) Press “Browse” and find the file you would like. Find the file you want. In this example it is an xyz file.<br />
| <br /><br />
|-<br />
| Once A file is picked then press “Load” in the Molecule File Browser window.<br />
| <br /><br />
|-<br />
| Going back to VMD Main<br />
| <br /><br />
|-<br />
|<br />
Press “Graphics” and Representations to open a long slender screen.(VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c8cfeb66ca7723b0.png|225x487px]]<br />
|-<br />
| Press the “Create Rep” Button (Graphical Representation window)<br />
| <br /><br />
|-<br />
| With one of the “Reps” highlighted in green, in the drawing Method switch from Lines to VDW<br />
| <br /><br />
|-<br />
|<br />
Change the Sphere scale to 0.2.<br />
<br />
Resolution can stay at default or can be raised if desired.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6ceb4d7e0c7c142a.png|157x340px]]<br />
|-<br />
|<br />
Click on the other line that still says “LINES” and in the Drawing Method change it to Dynamic Bonds<br />
<br />
Change Distance Cutoff to 1.6<br />
<br />
Bond Radius to 0.1<br />
<br />
And Resolution can stay at default or be increased<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_dba6c2b704e3401a.png|158x337px]]<br />
|-<br />
| Go back to VMD Main<br />
| <br /><br />
|-<br />
| Go to “Extensions,” “Visualizations,” and “Movie maker”<br />
| <br /><br />
|-<br />
| Go to the VMD Movie Generator window<br />
| <br /><br />
|-<br />
| In the box of “Name of movie” type the TDESMD_##_### you have.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|214x167px]]<br />
|-<br />
| “Set Working Directory“ choose the destination directory you want the movie to go to.<br />
| <br /><br />
|-<br />
| “Rotation Angle” I put it to 0 but you can put it to whatever.<br />
| <br /><br />
|-<br />
| “trajectory step size” 1<br />
| <br /><br />
|-<br />
| “Movie Duration” This depends on your frames from the (VMDMain) Usually 1000 frames is about 20-30 seconds if the speed (step) is set to 1. But By changing the speed of the video we should make the total video length to 30-45 seconds long.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|232x181px]]<br />
|-<br />
| Back to VMD Main window<br />
| <br /><br />
|-<br />
| Press “Mouse,” “Label,” “Bonds”<br />
| <br /><br />
|-<br />
| click on two neighboring atoms to get the atom name and bond length<br />
| <br /><br />
|-<br />
|<br />
IN VMD Main, go to “Graphics” and “Labels.” select “atoms” in the drop down menu on the “Labels” window.<br />
<br />
Select all atoms by clicking on the top atom and then “SHIFT+left click” the bottom atom to select all atoms<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
While all atoms are selected (highlighted in green). Go to “Properties” and in the “Format” box type %e%i.<br />
<br />
(The %e gives the element and the %i gives the index number)<br />
<br />
Some information can be presented by simply having your cursor over the %e%i and a yellow box will pop up for some info.<br />
<br />
You can move the lettering of each atom by moving the offset information.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_46eab8f828aa40bc.png|215x201px]]<br />
|-<br />
| Click the global properties and change text size and text thickness. ( all of this can be see in the (VMD1.9.3 OpenGL Display) [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_260ebe13a2eda05.png|162x114px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3968014bc41ed6a4.png|224x212px]]<br />
|-<br />
| in VMD movie Generator press “make movie” and quickly in VMD main press the play button.<br />
| <br /><br />
|-<br />
|<br />
Once the video is done a popup should be produced asking where to find videomach. Here is the directory for my computer.<br />
<br />
C:\Program Files (x86)\VideoMach\videomach.exe<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Once you put in the file videomach.exe then videomach should show up.<br />
| <br /><br />
|-<br />
| Switch the file type to MP4<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6a986dcfd4344618.png|374x27px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_da932791910b2a04.png|167x188px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d35497678abed954.png|155x364px]]<br />
|-<br />
| <br /><br />
| Polymer/Monomer Photodegradation: TDESMD 91 121 Bond Data<br />
|-<br />
| '''Obtaining MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| We will be using python scripts that are premade. In PHOTON<br />
| <br /><br />
|-<br />
| go to the TDESMD_###_### directory you want to go to<br />
| <br /><br />
|-<br />
| in here make another directory called massspectrum<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| mkdir massspectrum<br />
| <br /><br />
|-<br />
| cd masssspectrum<br />
| <br /><br />
|-<br />
| cp /Location/movieALL.xyz<br />
| This should be from the TDESMD_###_### you are in.<br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieALL.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Record number of molecules converted.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| python3 script1n.py<br />
| <br /><br />
|-<br />
| enter number of molecules converted<br />
| <br /><br />
|-<br />
| <br /><br />
| a frequency file should be created<br />
|-<br />
| cp frequency.txt frequency1.txt<br />
| <br /><br />
|-<br />
| awk 'ORS=(!(NR%2)?&quot;\n&quot;:&quot; &quot;)' RS=&quot; &quot; frequency.txt &gt; file1.txt<br />
| <br /><br />
|-<br />
| cp rmovingfile.sh<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
|-<br />
| bash rmovingfile.sh<br />
| <br /><br />
|-<br />
| vi file1.txt<br />
| Edit the file by deleting everything that doesn’t relate to the numbers.<br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| before [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b3731e92878413d2.png|289x328px]]<br />
|<br />
after<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c6aa114dd0a5843f.png|108x333px]]<br />
|-<br />
|<br />
IT IS NECESSARY TO MENTION THAT WHEN CREATING THE FRAGMENTS FILE AN EXTRA BLANK LINE WILL BE GENERATED. IT IS ESSENTIAL TO DELETE.<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_2104ec24e9b14aae.png|258x363px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_36d7bfd4ff6b9e13.png|218x371px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp file1.txt fragments<br />
| <br /><br />
|-<br />
| wc fragments<br />
|<br />
record the first number<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
in my case it is 21<br />
|-<br />
| ~/bin/DRESS_FRAGMENT_v2<br />
| <br /><br />
|-<br />
|<br />
21<br />
<br />
100<br />
<br />
300<br />
<br />
.1<br />
<br />
1<br />
<br />
62<br />
<br />
160<br />
<br />
<br /><br />
|<br />
number of states<br />
<br />
Min MASS (can change if desired)<br />
<br />
Max MASS (can change if desired)<br />
<br />
width of the line used<br />
<br />
magnitude (keep at one)<br />
<br />
number of atoms<br />
<br />
number of electrons<br />
|-<br />
| '''Plotting FERWE MASSPECTRUM'''<br />
| <br /><br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| e in the directory of the file you want to plot, in this example we are using MASS_SPEC.norm as our file with the data we want.<br />
|-<br />
|<br />
set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Monomer H2O Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br />To press the right mouse button and move mouse to preferred window size. Then left click<br />
| monomer h20<br />
|-<br />
| ressing p takes you to the previous zoom level, n to the next level, and u unzooms<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| <br /><br />
|-<br />
| '''Obtaining Multiple Cell MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| Make sure TDESMD CONTCAREXICTED and CONTCARGROUND files have been generated. Start in Cori go to Photon<br />
| <br /><br />
|-<br />
|<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test.sh<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ test_o.sh<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ANA_v2.exe<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/input_ANA<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/CONTCAREXICTED and GROUNDFILES<br />
|<br />
Take these files from Cori and put them into photon.<br />
<br />
These two files are needed for running the scripts properly<br />
<br />
These are the “important parts of the code”<br />
<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
input_ANA should have similar format to this<br />
<br />
Where the first 3 numbers are from POSCAR, the last 3 are the number of cells in the x,y,z axes respectively. This files was duplicated across Z axis.<br />
<br />
18.54438999999999<br />
<br />
14.45937999999995<br />
<br />
17.76793999999994<br />
<br />
1<br />
<br />
1<br />
<br />
10<br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
I used input_ANA<br />
|-<br />
|<br />
CHECK to see if OUTPUT has the correct number of atoms<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_4a92367ce10afd50.png|624x64px]]<br />
| Single cell had 620 atoms, I multipled it by 10 since I wanted to duplicate 10 times. If this is good procede to next steps<br />
|-<br />
| vi test.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_19d33c90b0cefb7a.png|278x215px]]<br />
| change the number to later stage in your trajectory(or the number where you think degradation had occurred. For example my total CONTCARGround and excited was about 191 files. I saw degradation occur later in the simulation so I chose the number 190. This changes with the simulations.<br />
|-<br />
| bash test.sh<br />
| this runs the test.sh file<br />
|-<br />
| bash test_o.sh<br />
| <br /><br />
|-<br />
| NOW LOOKING INTO PHOTON<br />
| <br /><br />
|-<br />
| cp all files into photon directory<br />
| <br /><br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieANA.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
Record number of molecules converted.<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| Follow regular mass spec steps from above. Rather than using MovieALL we are using movieANA.xyz.<br />
| <br /><br />
|-<br />
| <br /><br />
|<br />
photon<br />
<br />
/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
<br />
cori<br />
<br />
/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test<br />
|-<br />
| plummer code for renaming files<br />
| /global/project/projectdirs/m1251/vasp/CHEM676/meade/MONOMER_H2O/TDESMD_162_167/massspec/multiplemassspectest/plumoutput<br />
|-<br />
| '''How to read/get the fragments with their xyz coordinates'''<br />
| <br /><br />
|-<br />
|<br />
Go into the directory with the shot files with the multiple cells. Make sure that some files have this comment inside of them by opening a shot file that has name shot##n#.xyz<br />
<br />
The comment should be.<br />
<br />
created by duplicating cells#<br />
| <br /><br />
|-<br />
| grep -E 'created by duplicating cells.* 17.' shot*<br />
|<br />
finds lines that have both “created by duplicating cells” and the number 17 in the same line.<br />
<br />
17 is the mass number we are looking for<br />
<br />
I am unsure what the “.*” does in this code<br />
<br />
make sure there is a space between the * and the 17<br />
|-<br />
| Then open one of the xyz files listed.<br />
| <br /><br />
|-<br />
| Literally drag and copy the file information into a notepad document in your desktop.<br />
| <br /><br />
|-<br />
| more shot291n10.xyz<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_9c48a58b9e24dc27.png|210x268px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_7c3a89906ed25e30.png|180x270px]]<br />
|-<br />
| open the xyz file in Avogadro and save as mol file.<br />
| <br /><br />
|-<br />
| the Avogadro file is correct visualization of the fragments. Mol file will have some defects so you will have to change those.<br />
| <br /><br />
|-<br />
| Making Video from output of VASP<br />
| <br /><br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exemore<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
I used input_ANA<br />
|-<br />
| '''LAAMPS'''<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_796f3aade1e5b306.png|314x150px]]<br />
|<br />
testwater file:<br />
<br />
Epsolon is Inputs from something like DFT<br />
<br />
<br />
Columns:<br />
<br />
Atom ID<br />
<br />
Molecule ID<br />
<br />
Atom Type<br />
<br />
Q - charge<br />
<br />
X<br />
<br />
Y<br />
<br />
Z<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_f474cba5e65bdaab.png|306x66px]]<br />
| This infor can be put in input script if desired.<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_a9802b534de46367.png|624x180px]]<br />
|<br />
bond id<br />
<br />
bond tuype<br />
<br />
atom id that connects<br />
<br />
atom id that connects as well<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3b00d172c3a15f2b.png|310x103px]]<br />
|<br />
Angle ID<br />
<br />
Angle Type 1 being single bond 2 being double, etc<br />
<br />
Item ID such as first atom O , 2<sup>nd</sup> and 3<sup>rd</sup> atom H<br />
|-<br />
| qsub submit.sh<br />
| submits the job using submit.sh<br />
|-<br />
|<span style="background: #101010">qstat -u meade.erickson</span><br />
| shoes me my jobs<br />
|-<br />
| <span style="background: #101010">chmod -777 lmp_mpi_12Dec18</span><br />
| modifies permission for this file<br />
|-<br />
| [meade.erickson@login0002 testwater]$ chmod -777 ~/bin/lmp_mpi_12Dec18<br />
| giving permission while in a different<br />
|-<br />
| topo readlammpsdata project.data<br />
| reads data file for vmd tk console<br />
|-<br />
| Inputting files, put in .dcd file then right click on it and “Load Data into Molecule” then select .data file.<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_73b2551819922e26.png|287x164px]]<br />
| <br /><br />
|-<br />
| module avail<br />
| Available software in ccast<br />
|-<br />
| qdet job id<br />
| qdel ######<br />
|-<br />
| fragment 0 to 9 or fragment 14<br />
| <br /><br />
|}</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=VASP_and_VMD_on_CCAST&diff=129VASP and VMD on CCAST2022-10-21T17:45:45Z<p>Sysadmin: removed top info</p>
<hr />
<div>__TOC__<br />
<br />
{|<br />
|-<br />
|<br />
'''Misc. Tools'''<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| cd foldername<br />
| lets you go into a folder<br />
|-<br />
| cd ..<br />
| Backs you out of folder you are in<br />
|-<br />
| cd -<br />
| Goes back to previous directory<br />
|-<br />
| cd ~<br />
| Goes to home directory<br />
|-<br />
| freenodes<br />
| in ccast<br />
|-<br />
| vi filename<br />
| Lets you go into file to edit<br />
|-<br />
| i<br />
| When in “vi” this initiates the editing<br />
|-<br />
| :wq<br />
| This lets you stop editing<br />
|-<br />
| “Ctrl+c” press on keyboard, not a typing thing<br />
| Lets you stop any current running process<br />
|-<br />
| pwd<br />
| Gives you current directory location<br />
|-<br />
| wc<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| ls<br />
| Lists what is in the current directory<br />
|-<br />
| ls -lt<br />
| Lists what is in the current directory sorting from newest to oldest<br />
|-<br />
| ls –lt W*<br />
| Lists everything that starts with a W sorting from newest to oldest. (Can be used for other letters as well)<br />
|-<br />
| ls -lisanh<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisanh filename<br />
| gives information on the specific file<br />
|-<br />
| ls -lisan<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisan filename<br />
| gives information on the specific file<br />
|-<br />
| ls -a<br />
| shows all files, including hidden ones<br />
|-<br />
| babel –ixyz file.xyz –opdb file.pdb xyz to pdb<br />
|<br />
Using babel, converts the input file of .xyz to outputfile of .pdb<br />
<br />
That is where the –ixyz and –opdb comes from<br />
|-<br />
| “shift+alt+left click”<br />
| For slecting just comlumn or just row information you want exactly<br />
|-<br />
| egrep '163|166' OS_STRENGTH<br />
| grep technique<br />
|-<br />
| more filename<br />
| Lets you see what is in the file<br />
|-<br />
| more ~/bin/filename<br />
| Lets you see the file in a location not in your current directory.<br />
|-<br />
| more filename | grep ### or value | grep again if desired<br />
| shows file and grabs lines with same input you put after the grep command<br />
|-<br />
| cp /Location/filename .<br />
| Copies a file from a different directory location and input it into the current directory. Don’t forget the PERIOD!<br />
|-<br />
| P /Location/filename ./newfilename<br />
| Copies a file from a different directory location and input it into the current directory and rename it in one step<br />
|-<br />
| scp sourceUser:/sourceLocation/filename .<br />
| Takes file from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/* .<br />
| Takes ALL files from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/*.filetype .<br />
| Takes ALL files with that specific file type from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp * destinationUser:/destinationLocation<br />
| Sends all files from source directory to destination directory. Must be in source directory when typing this line<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/DIMabsorptionspec<br />
| Example of scp<br />
|-<br />
| sbatch corifilename<br />
| Lets you submit a file in cori using a corifile made by others<br />
|-<br />
| grep specificdetail filename<br />
| Grabs the specific detail from the file you chose<br />
|-<br />
| sort -n -r filename<br />
| sorts the file in ascending order<br />
|-<br />
| babel BondLength.xyz -O out.xyz --separate<br />
| THIS IS CORRECT WAY THE SEP WORD IS SPELLED INCORRECTLY BUT IS CORRECT FOR BABEL<br />
|-<br />
| awk '{ if($4 &gt;= 4.0000 &amp;&amp; $4&lt;=5.0000) print }' OS_STRENGTH1<br />
| lets you get lines of data that contain the range from 4-5 from column 4<br />
|-<br />
| users<br />
| lets you see what users are on<br />
|-<br />
| clear<br />
| lets you clear screen<br />
|-<br />
| who<br />
| who is on and more detail about them<br />
|-<br />
| w<br />
| all the people and differing details about them<br />
|-<br />
| id<br />
| lets me see what groups I am in<br />
|-<br />
| freenodes<br />
| lets you see if nodes are available or not<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6335cce9234bc4ef.png|439x225px]]<br />
|<br />
|-<br />
| pbsnodes –avSj | sort -n<br />
| shows available nodes, the | sorts it into order<br />
|-<br />
| pbsnodes –avSj | sort –n | more<br />
| does this in page format rather than all at once<br />
|-<br />
| module avail<br />
| what software is available<br />
|-<br />
| $HOME<br />
| lets you go to your login directory “home” directory<br />
|-<br />
|<br />
thunder.ccast.ndsu.edu<br />
<br />
meade.erickson<br />
| to get into putty<br />
|-<br />
| C:\Users\meade.erickson\Documents\OriginLab\User Files\<br />
| Origin themes location<br />
|-<br />
| '''Explanations'''<br />
| <br /><br />
|-<br />
| fermi<br />
| fermi energy is referring to the energy difference between the highest and lowest occupied single-particle states. ([https://en.wikipedia.org/wiki/Fermi_energy <u>https://en.wikipedia.org/wiki/Fermi_energy</u>])<br />
|-<br />
| '''Explanations Input Files:'''<br />
| <br /><br />
|-<br />
| INCAR<br />
| It is the central input file of VASP. It determines &quot;what to do and how to do it&quot;. ([https://www.vasp.at/wiki/index.php/INCAR <u>https://www.vasp.at/wiki/index.php/INCAR</u>])<br />
|-<br />
| POSCAR<br />
| <br /><br />
|-<br />
| POTCAR<br />
| <br /><br />
|-<br />
| KPOINTS<br />
| <br /><br />
|-<br />
| '''Explanations Output Files:'''<br />
| <br /><br />
|-<br />
| CONTCAR<br />
|<br />
Contains actual coordinates, velocities, and predictor corrector coordinates. 1<sup>st</sup> block is lattice parameters and atom coordinates. 2<sup>nd</sup> block Initial Velocites for atoms. 3<sup>rd</sup> block Predictor-corrector coordinates.<br />
<br />
Written at end of each job. Used as the POSCAR file for next MD job.<br />
|-<br />
| WAVECAR<br />
| Provides wavefunctions. we usually use it for spectrum data ([https://www.vasp.at/wiki/index.php/WAVECAR <u>https://www.vasp.at/wiki/index.php/WAVECAR</u>])<br />
|-<br />
| OS_STRENGTH<br />
| Information about the OS_STRENGTH<br />
|-<br />
|<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8bf47c7304419724.png|354x154px]]<br />
|<br />
From Wikipedia<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
In spectroscopy, '''oscillator strength''' is a dimensionless quantity that expresses the probability of [https://en.wikipedia.org/wiki/Absorption_(electromagnetic_radiation) <u>absorption</u>] or [https://en.wikipedia.org/wiki/Emission_spectrum <u>emission</u>] of [https://en.wikipedia.org/wiki/Electromagnetic_radiation <u>electromagnetic radiation</u>] in transitions between [https://en.wikipedia.org/wiki/Energy_level <u>energy levels</u>] of an atom or molecule<sup>[</sup>[https://en.wikipedia.org/wiki/Wikipedia:Accuracy_dispute#Disputed_statement <sup>''<u>dubious</u>''</sup>]<sup>'' – ''</sup>[https://en.wikipedia.org/wiki/Talk:Oscillator_strength#Wrong_definition_oscillator_strength <sup>''<u>discuss</u>''</sup>]<sup>]</sup>.<sup><u>[https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Demtr%C3%B6der2003-1 [1]][https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Robinson1996-2 [2]]</u></sup> The oscillator strength can be thought of as the ratio between the quantum mechanical transition rate and the classical absorption/emission rate of a single electron oscillator with the same frequency as the transition.<br />
|-<br />
| POTIM (in INCAR file)<br />
| timestep in femtoseconds. So POTIM=1 is 1 femtosecod<br />
|-<br />
| '''Density of States'''<br />
| <br /><br />
|-<br />
| To get DOS you need to run the correct calculations in VASP using 4 files. INCAR, POSCAR, POTCAR, OUTCAR<br />
| <br /><br />
|-<br />
| Once you get output from your submitted information to cori or photon you can follow these steps to get Density of states<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| Grabs the number of bands (we need this for later) (STATES)<br />
|-<br />
| vi input_overlap<br />
| editing input_overlap<br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| Number of Bands from the grep NBA OUTCAR line<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| quitting editing input_overlap<br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| getting energy pop from different location<br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| wc STATES<br />
| <br /><br />
|-<br />
| grep NELE OUTCAR<br />
| record number of electrons<br />
|-<br />
| grep NIO OUTCAR<br />
| record number of ions<br />
|-<br />
| grep E-fer OUTCAR<br />
| record the three values and their names. If this line doesn’t work the try next line)<br />
|-<br />
| grep ferm OUTCAR | tail –n 1<br />
| record the three values and their names<br />
|-<br />
| ~/bin/DRESS_DOS_NORM<br />
| Executes the DOS code from different file location<br />
|-<br />
|<br />
Number of states from grep NBA OUTCAR<br />
| grep NBA OUTCAR line<br />
|-<br />
| -20<br />
| <br /><br />
|-<br />
| 20<br />
| <br /><br />
|-<br />
|<br />
E-fermi info from grep E_fermi<br />
<br />
<br /><br />
|<br />
To literally line up DOS and STATES, you may need to type in 0 for both fermi and alpha+beta.<br />
<br />
<br />A more standard way is to use the middle point energy between HOMO and LUMO i.e. (E_HOMO+E_LUMO)/2 for fermi and 0 for alpha+beta. In this way, the mid point of DOS is at zero, anything below zero is occupied and above zero is unoccupied.<br />
|-<br />
|<br />
alpha+beta info from grep E_fermi<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| .01<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
|<br />
Number of ions from grep NIO OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
Number of electrons from grep NELE OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| 3<br />
| <br /><br />
|-<br />
|<br />
gnuplot ~/bin/gnuprog_DOS.fermi<br />
<br />
<br /><br />
| Plotting the DOS (This is done in NERSC or PHOTON)<br />
|-<br />
|<br />
ps2pdfwr DOS.ps<br />
<br />
<br /><br />
| Converting DOS file from .ps to .pdf<br />
|-<br />
|<br />set out MEADESDOS.ps MAY NOT NEED THIS NOT SURE YET<br />
set style line 4 lw 4 lc -1; unset key; plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
set style line 4 lw 4 lc -1<br />
<br />
unset key<br />
<br />
plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4;<br />
<br />
set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0<br />
<br />
set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0<br />
<br />
set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0<br />
<br />
set tics font &quot;,30&quot;<br />
<br />
set ytics offset 0,0,0<br />
<br />
set xtics offset -0.4,-1,0<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.518567):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.518567):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.173350):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.17335):4 w l title &quot;Wet Monomer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Wet Monomer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
|<br />
I am initially specifying a style that I will be using. In this case chose style 4 and am editing it to what I want.<br />
<br />
lw stands for linwidth, lc line color, not sure where the library is but lc -1 is black, 0=red I think and find out other through trial and error<br />
<br />
lw is just multiplyer, so lw 4 is 4 times larger than default.<br />
<br />
unset key gets rid of key,<br />
<br />
[#,#] is the range of my x axis graph.<br />
<br />
($#-###): $# is the column I want to take data from which is being subtracted by the amount of shift I am giving to the data (we want data to be centered at 0 sometimes and so for DOS we take the middle between the start of non occupied plus the end of occupied and divide by 2 to get our shift. (Notoccupied + occupied)/2 ) and the number after the “:” is the column I am correlating the X and Y axis with. So when we plot stuff we need to specify what is the x and what is the y axis.<br />
<br />
not sure exactly what the “w i” portion does but it helps us change from points to lines and then is what fills in the red portion of our graph. (The “i” stands for impulses. Add an extra “lw 2’ to increase the line width as shown in green text.) “w l” does the same thing but DOESN”T fill out the the electron density portion.<br />
<br />
xlabel and y labels are the axis titles and offsets help position them due to the changing sizes of the axes and tics.<br />
<br />
Tics are the numbers on the Y and x axis so ytics offsets them and xtics does the horizontal axis change.<br />
<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
| '''Absorption Spectrum - Transition Energy eV – Wavelength nm'''<br />
| <br /><br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
|<br />
copying INCAR-gm into the directory as long as you are in it<br />
<br />
MAKE SURE<br />
<br />
LWAVE= .TRUE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori07:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec/TEST<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Or make sure the INCAR has this Ag+Carboxyl<br />
<br />
#ISIF=7<br />
<br />
#PREC = HIGH<br />
<br />
ENCUT= 300<br />
<br />
IBRION =3<br />
<br />
POTIM=0.2<br />
<br />
NSW = 30000<br />
<br />
ALGO = NORMAL #48<br />
<br />
ISMEAR = 0<br />
<br />
SIGMA = 0.01<br />
<br />
LREAL = A<br />
<br />
#ISTART = 0<br />
<br />
LWAVE = .TRUE.<br />
<br />
LCHARG = .FALSE.<br />
<br />
EDIFF=1E-6<br />
<br />
EDIFFG=-0.05<br />
<br />
LVTOT = .FALSE.<br />
<br />
#LVDW=.TRUE.<br />
<br />
#SPRING = -5<br />
<br />
#LCLIMB = .TRUE.<br />
<br />
#IMAGES = 7<br />
<br />
#IOPT=7<br />
<br />
#ICHAIN=0<br />
<br />
NELM=40<br />
<br />
#ISPIN=2<br />
<br />
ISYM=0<br />
<br />
NPAR=4<br />
<br />
<br /><br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|345x215px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| It is necessary to have the ‘spectrum’ file made which comes from a healthy WAVECAR which is used to generate OS_STRENGTH. After OS_STRENGHT we make the spectrum ~/bin/SPECTRUM_imp3 file.<br />
| <br /><br />
|-<br />
| Assuming you have conducted a calculation which generated a healthy WAVECAR file.<br />
| <br /><br />
|-<br />
| If healthy WAVECAR exists then…<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| record Number of bands<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
|<br />
1<br />
<br />
Number of Bands<br />
<br />
1<br />
| 36740210<br />
|-<br />
|<br />
‘esc’ key<br />
<br />
:wq<br />
| <br /><br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| <br /><br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| ./osc_str.exe<br />
| This makes the OS_STRENGTH (oscillator strength)<br />
|-<br />
| grep '2.0000 0.0000' OS_STRENGTH &gt; OS_STRENGTHvc<br />
|<br />
Takes the orbital pairs from 2.0000 to 0.0000 into new file.<br />
<br />
<br /><br />
|-<br />
| cp OS_STRENGTH OS_STRENGTH1<br />
| making sure I have original OS_STRENGTH<br />
|-<br />
| cp OS_STRENGTHvc OS_STRENGTH<br />
| making sure I have file OS_STRENGvc kept for 2.0000 – 0.0000 but it will be our OS_STRENGTH<br />
|-<br />
| wc OS_STRENGTH<br />
| Record First number<br />
|-<br />
| ~/bin/SPECTRUM_imp3<br />
| <br /><br />
|-<br />
|<br />
Number from OS_STERNEGTH<br />
<br />
.1<br />
<br />
10<br />
<br />
.001<br />
<br />
HOMO<br />
|<br />
Should be able to find HOMO from STATES file (or energy_pop)<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Can change the “.001” to “0.1 or 0.01” to smooth out the line<br />
|-<br />
| a ‘spectrum’ file should have been made. This holds all the spectrum data. The spectrum can be visualized using these commands in Cori. If you want to visualize in photon then use the gnuplot information below<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE<br />
|<br />
IN CORI YOU CAN USE THESE<br />
<br />
THEN SCP them to photon, then the desktop!<br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE_nm_200_800<br />
|<br />
|-<br />
| ps2pdfwr SPE.ps<br />
|<br />
|-<br />
| ps2pdfwr SPE_nm.ps<br />
|<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec<br />
| example make sure you are in the directory of the files you want to send. Send it to a premade directory in photon<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
|<br />
FOR PHOTON USE THESE STEPSWe are plotting the information of spectrum to get the spectrum in wavelength of nanometers nm and transition energy in electron volts eV<br />
| <br /><br />
|-<br />
| more spectrum<br />
|<br />
(Be sure to be in correct directory)<br />
<br />
Column 1 is the spectrum in wavelength of nm<br />
<br />
column 2 is the spectrum in transition energy of eV<br />
<br />
column 3 is absorbance/absorption in a.u. (absoption units). Absorbance goes with wavelength, absorption goes with transition energy<br />
<br />
column 4 idk what this is.<br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| plot ‘spectrum’ 2:3<br />
| This gives you transition energy in x axis and absorption in a.u for yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [0:10] 'spectrum' using 2:3 w l title &quot;Transition Energy&quot; ls 4; set xlabel 'Transition Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorption a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| This is transition energy in x axis and absorption in yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [200:800] 'spectrum' using 1:3 w l title &quot;Absorption Spectrum&quot; ls 4; set xlabel 'Wavelength, nm' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorbance, a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| nm in x axis and absorbance in yaxis<br />
|-<br />
| '''UV-Vis Absorption Spectrum'''<br />
| <br /><br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
We did one calculation already but we need to do a “geometry optimization” for the next one.<br />
<br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
| copying INCAR-gm into the directory as long as you are in it<br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|352x219px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| '''Generating CANDIDATES file for Photofragmentation'''<br />
| <br /><br />
|-<br />
| This is after calculations have been ran for MD???<br />
|<br />
Watch this first‼ Part 2 of Dr.Kilins video for how to generate the CANDIDATES file.<br />
<br />
[https://www.youtube.com/watch?v=NIrs1yXJj1Y&feature=youtu.be <u>https://www.youtube.com/watch?v=NIrs1yXJj1Y&amp;feature=youtu.be</u>]<br />
|-<br />
| We want to get information from the WAVECAR that can help us determine the “electron hopping” so we have to find the OS_STRENGTH (oscillator strength) that is high. Since higher OS_Strength the higher the probability that given pair of orbitals is excited. STATES file should have been generated before I think.<br />
|<br />
Watch this second‼ Part 1 of Dr. Kilins video for what to do after CANDIDATES file was generated and explains what is in it.<br />
<br />
[https://www.youtube.com/watch?v=A8BXT-FnuxM&t=212s <u>https://www.youtube.com/watch?v=A8BXT-FnuxM&amp;t=212s</u>]<br />
|-<br />
| grep –A 700 fermi OUTCAR | tail –n 700 &gt; STATES1<br />
| Fermi information from OUTCAR into a file called STATES 1. Not sure what the –A or 700 mean. But the tail is choosing a specific section of the data, the “tail” end of the chosen information from the OUTCAR file.<br />
|-<br />
| more STATES1<br />
| Determine HOMO and LUMO, HOMO is the highest orbital with a 2, LUMO is the first 0. In my case HOMO is 112 and LUMO is 113<br />
|-<br />
| NOT SURE HOW HE GENERATED STATES FILE<br />
| <br /><br />
|-<br />
| head STATES<br />
| Record the lowest number, in my case it is 81 for DIMER<br />
|-<br />
| tail STATES<br />
| Record the Highest number, in my case it is 160 for DIMER<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| 160<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| more ~/bin/ADVICE.spectrum<br />
| This will make spectrum. This will show 3 lines of code. Literally copy one line and press enter starting with the “module swap PrgEnv-intel PrgEnv-gnu”<br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| copying templates and inputting them into the directory you are in<br />
|-<br />
| ./osc_str_CHEM676.exe<br />
| This reads binary WAVECAR file, Reads input_overlap, creates Oscillator strength<br />
|-<br />
| ls –lt | head<br />
| This should show a fresh OS_STRENGTH was made<br />
|-<br />
| grep ‘2.0000 0.0000’ OS_STRENGTH &gt; OS_NEW<br />
| In the OS_STRENGTH file we will see lines of no transition from occupied to unnoccupied. What we want to see is the lines that have an occupied originally (2.0000) and then are unoccupied (0.0000). This shows excitation. The lines that meet this requirement are then inputted into OS_NEW<br />
|-<br />
| head OS_NEW<br />
| This should show a range of orbitals that have been occupied and are now unoccupied. With the differing orbitals involved. Columns 1 and 2 are the initial and final orbitals that are involved with the “hopping” of electrons. The 3rd column is oscillator Strength. 4th Column is Transition energy.<br />
|-<br />
| mv OS_NEW OS_STRENGTH<br />
| We don’t need all OS_STRENGTH data, just the stuff we “grepped” in OS_NEW. This overrides OS_STRENGTH with information from OS_NEW.<br />
|-<br />
| wc OS_STRENGTH<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| head –n 30 OS_STRENGTH<br />
| In this particular case we only want the top 30 rather than the original value the first column gave was. In my case it was about 1536.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4<br />
| To sort in ascending order of transition energy. 4 is the 4<sup>th</sup> column. I don’t know what –gk means.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4 &gt; CANDIDATES<br />
| Inputs this information into CANDIDATES file<br />
|-<br />
| '''Orbital Visualization'''<br />
| <br /><br />
|-<br />
| I don’t have all information available here, for example I personally didn’t make the PARCHG files.<br />
| <br /><br />
|-<br />
|<br />
Kilins Guidance to make PARCHG files… I think<br />
<br />
Dear Meade,<br />
<br />
below are steps I did for creating orbitals files on nersc.<br />
<br />
I will let you know when they are ready and instruct on how to visualize.<br />
<br />
slides are good, under given limits.<br />
<br />
In the given time, you may add conclusions as you see them..<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; mkdir ORBITALS<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp WAVECAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp CONTCAR ORBITALS/POSCAR<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp POTCAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp *.sh ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp KPOINTS ORBITALS/<br />
<br />
&gt; cd ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; grep NBA ../OUTCAR<br />
<br />
k-points NKPTS = 1 k-points in BZ NKDIM = 1 number of bands NBANDS= 160<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; cp ~/bin/INCAR/INCAR-pc ./INCAR<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; tail INCAR<br />
<br />
IBRION=2 #conjugate-gradient algorithm used to relax ions (bad<br />
<br />
NSW=0 #number of ionic steps<br />
<br />
POTIM= .2 #time step in fs<br />
<br />
EDIFFG=-0.001 #minimum energy difference between ionic iterations<br />
<br />
EINT= -4.4 -0.5<br />
<br />
EINT= -6 2 <br />
<br />
LVTOT = .TRUE.<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; sbatch cori_debug.sh <br />
<br />
Submitted batch job 32413471<br />
<br />
many thanks for the new version!<br />
<br />
The main question in your email is orbitals with VASP:<br />
<br />
a. generate or use premade healthy WAVECAR <br />
<br />
(it is an analog of .chk in gaussian)<br />
<br />
b. in a new directory, use template of the INCAR for orbitals and run vasp once again.<br />
<br />
the templates are named ~/bin/INCAR/INCAR_pc and similar names.<br />
<br />
(it is an analog of the cubegen procedure in gaussian)<br />
<br />
c. upon successful run, there appear files nammed PARCHG.XXXX.ALLK, where XXXX stands for the index of orbital, e.g. PARCHG..01112.ALLK<br />
<br />
(those files are analogs of the .cube files in gaussian. note that vasp DOES not generate literal .cube files!)<br />
<br />
d. download these files and open then in the VMD - the same software you have used for the trajectories.<br />
<br />
I will have a look on the slides and let you nko if any quicl changes are needed or we can go ahead with this version<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
<br /><br />
|<br />
|-<br />
| Step 2. Changing PARCHG Files to cube files (For Orbital Visualization)<br />
| From what I have seen we need cube files to show in guassview AND VMD<br />
|-<br />
| Once obtained PARCH.####.ALLK Files then use following command in PHOTON to convert the orbitals you care about to cube!<br />
| <br /><br />
|-<br />
| ~/vtstools/chg2cube.pl filename<br />
| <br /><br />
|-<br />
| 3<br />
| selecting PARCHG<br />
|-<br />
| 1 6 7 8<br />
| (These are the atomic numbers for the atoms in your system. In my case I have Hydrogen, Nitrogen, Oxygen, and Carbon<br />
|-<br />
| Your cube file is then generated<br />
| <br /><br />
|-<br />
| SEND cube files, POSCAR (OR CONTCAR), CHGCAR FILES TO DESKTOP<br />
| <br /><br />
|-<br />
| Step 3. Using VMD to generate orbitals<br />
| <br /><br />
|-<br />
| Open VMD<br />
| <br /><br />
|-<br />
|<br />
In the VMD Main window go to “File” “new Molecule” Upload your<br />
<br />
1.POSCAR (OR POSSIBLY CONTCAR): By selecting VASP_POSCAR under the “Determine file type:”<br />
<br />
2. CHGCAR: By selecting VASP_CHGCAR under the “Determine file type:”<br />
<br />
3. USE THIS ONE PARCHG: By selecting VASP_PARCHG under the “Determine file type:” (Even though the file ends in .cube we are classifying it as PARCHG)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b0283f3c67e04e7d.png|219x134px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_eae9e77b87b8627b.png|222x103px]]<br />
|-<br />
|<br />
After all three are uploaded,<br />
<br />
in the VMD Main window go to “Graphics” and then “Representations”<br />
| <br /><br />
|-<br />
|<br />
In the Graphical Representations window follow these three examples.<br />
<br />
<br />(Change the file you are using by the dropdown menu of “selected Molecule”)<br />
|<br />
The red lettering means that PARCHG Isosurface isn’t being shown. You can double click on the red lettering to make it black and showing on the OpenGL Display Window. Double click again to revert to black.<br />
<br />
Parchrg can be shown and can be visualized in conjungtion with the CHGCAR file<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_e7a558f4dc169c4e.png|121x260px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b15ffb6b911b765.png|124x266px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3734231f026ab64e.png|124x265px]]<br />
|<br />
|-<br />
|<br />
-You can change the iso value in CHGCAR to get varying orbital visualizations.<br />
<br />
-Play with it.<br />
<br />
-Change the color by changing the ColorID to different ones.<br />
<br />
-(Range in CHGCAR doesn’t matter)<br />
|<br />
For Monomer Project Show<br />
<br />
all index 52 47 29 44 38 28 18 56 64<br />
<br />
H52<br />
<br />
H47<br />
<br />
H29<br />
<br />
H44<br />
<br />
H38<br />
<br />
H28<br />
<br />
H18<br />
<br />
H56<br />
<br />
H64<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c5f67d67a9a89fa7.png|355x180px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d4c6c68549d8e025.png|357x209px]]<br />
| <br /><br />
|-<br />
| To get rid of opaque water molecuels make both of these in “Graphics” “Materials” 123 is for faded bods and 124 is for the orbitals<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8d905765b5183c3.png|142x229px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_449e5be8bb4a3541.png|139x222px]]<br />
|<br />
|-<br />
| For my Monomer_H2O first frame hide these atoms if you don’t want in view<br />
| <br /><br />
|-<br />
|<br />
O108<br />
<br />
O99<br />
<br />
O94<br />
<br />
O101<br />
<br />
O81<br />
<br />
O91<br />
<br />
O90<br />
<br />
O86<br />
<br />
O82<br />
<br />
O87<br />
<br />
O79<br />
<br />
<br />H73<br />
<br />
H45<br />
<br />
H62<br />
<br />
H51<br />
<br />
H58<br />
<br />
H50<br />
<br />
H48<br />
<br />
H14<br />
<br />
H22<br />
<br />
H36<br />
<br />
H42<br />
<br />
H31<br />
<br />
H20<br />
<br />
H23<br />
<br />
H35<br />
<br />
H30<br />
<br />
H54<br />
<br />
H59<br />
<br />
H25<br />
|<br />
all not index ### ### ### ###...<br />
<br />
all not index 108 99 94 101 81 91 90 86 82 87 79 73 45 62 51 58 50 48 14 22 36 42 31 20 23 35 30 54 59 25<br />
<br />
<br />backbone <br />
<br />
index 97 103 11 9 10 13 12 96 1 84 78 3 4 100 77 0 8 2 5 6 7<br />
|-<br />
| '''These are for the figure first frame for still seeing the water atoms but hazy'''<br />
| <br /><br />
|-<br />
|<br />
H22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46<br />
<br />
<br />O90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
| 22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46 90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
|-<br />
| '''Index for Diagram for pics Middle Frame for H20 molecules'''<br />
| <br /><br />
|-<br />
|<br />
H63<br />
<br />
H33<br />
<br />
H17<br />
<br />
H21<br />
<br />
60<br />
<br />
68<br />
<br />
42<br />
<br />
38<br />
<br />
34<br />
<br />
50<br />
<br />
72<br />
<br />
73<br />
<br />
16<br />
<br />
61<br />
<br />
55<br />
<br />
71<br />
<br />
59<br />
<br />
69<br />
<br />
40<br />
<br />
14<br />
<br />
39<br />
<br />
74<br />
<br />
67<br />
<br />
53<br />
<br />
49<br />
<br />
41<br />
<br />
52<br />
<br />
27<br />
<br />
15<br />
<br />
31<br />
<br />
23<br />
<br />
54<br />
<br />
62<br />
<br />
32<br />
<br />
64<br />
<br />
30<br />
<br />
26<br />
<br />
57<br />
<br />
37<br />
<br />
51<br />
<br />
58<br />
<br />
70<br />
<br />
43<br />
<br />
25<br />
<br />
35<br />
<br />
46<br />
<br />
45<br />
|<br />
O80<br />
<br />
87<br />
<br />
99<br />
<br />
81<br />
<br />
88<br />
<br />
105<br />
<br />
104<br />
<br />
91<br />
<br />
90<br />
<br />
106<br />
<br />
97<br />
<br />
108<br />
<br />
102<br />
<br />
89<br />
<br />
93<br />
<br />
101<br />
<br />
92<br />
<br />
107<br />
<br />
78<br />
<br />
109<br />
<br />
86<br />
<br />
83<br />
<br />
94<br />
<br />
63 33 17 21 60 68 42 38 34 50 72 73 16 61 55 71 59 69 40 14 39 74 67 53 49 41 52 27 15 31 23 54 62 32 64 30 26 57 37 51 58 70 43 25 35 46 45 80 87 99 81 88 105 104 91 90 106 97 108 102 89 93 101 92 107 78 109 86 83 94<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_783e625ce0b8a321.png|182x247px]]<br />
|-<br />
| '''Preparation of TDESMD Orbital Files and Job Submission'''<br />
| <br /><br />
|-<br />
| mkdir TDESMD_111_114<br />
| We will now use the information of the CANDIDATES file and create new folder for each scenario. You create a file titled using the orbitals involved. In my case the first scenario is with orbitals 111 and 114<br />
|-<br />
| cd TDESMD_111_114<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/INCAR_EXCITED<br />
| Taking INCAR_EXCITED file from location. In my case it is here.<br />
|-<br />
| vi INCAR_EXCITED<br />
| Edit the INCAR_EXCITED file to make it look like<br />
|-<br />
| i<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_79a32df6f9f33bb0.png|303x201px]]<br /><br />
|-<br />
| <br /><br />
|<br />
Change FERWE=#### to match the total number of orbitals/bands. In this example we have a total number of 160 bands/orbitals.<br />
<br />
<br />
The first segment of FERWE, “110*1” says we have the first 110 orbitals occupied. The second portion is orbital number 111 which we have listed as a “0” signifying this orbital is unnoccopied for our job submission. Then orbital number 112 is occupied with “1”. Then orbital number 113 is unoccupied with “0”. Then orbital number 114 is occupied with “1”. The last part of “46*0” is talking about the other orbitals not mentioned. These orbital should add up to 160. These numbers change with the differing orbitals designating in the CANDIDATES FILE.<br />
<br />
The main thing that is changing is the position of one occupied orbital and one unoccupied orbital when the compound is in ground state. In this example number 111 is usually occupied in ground state and we are moving the electron to 114 which is unoccupied in ground state.<br />
|-<br />
| <br /><br />
| ONE more example is as follows. With TDESMD_102_115<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_51e84877e4374ca6.png|300x199px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POSCAR ./CONTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/KPOINTS<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/debug1.sh<br />
| <br /><br />
|-<br />
| sbatch debug1.sh<br />
| <br /><br />
|-<br />
| '''Creating Videos from movieAll.xyz files and uploading them on youtube'''<br />
| <br /><br />
|-<br />
| scp –r TDESMD_###_### DestinationUser:/DestinationLocation<br />
|<br />
Starting in cori<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Using this to send files from cori to photon. We specifically want the movieALL.xyz files.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
This is sending the WHOLE directory and everything it contains<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| mkdir TDESMD_###_###<br />
|<br />
In DESKTOP<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
generate TDESMD_###_### directories<br />
|-<br />
| scp kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_##_###/movieALL.xyz .<br />
| WHILE IN THE TDESMD DIRECTORY THAT YOU WANT TO BE THE DESTINATION take “movieALL.xyz” from photon and put into specific directory<br />
|-<br />
| <br /><br />
| Once the file is in your desktop you can go to your windows folders and open it to check if it is there.<br />
|-<br />
| In windows search bar<br />
| <br /><br />
|-<br />
| VMD<br />
|<br />
type vmd in windows search bar and open it.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
VMD will open with multiple screens.<br />
<br />
<br /><br />
|-<br />
| We will focus on this screen first to open all the extra screens we need. (VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_5f31d31c2c72c623.png|244x114px]]<br />
|-<br />
| Press “File”, “New Molecule,”<br />
| <br /><br />
|-<br />
| On the window that comes up (Molecule File Browser) Press “Browse” and find the file you would like. Find the file you want. In this example it is an xyz file.<br />
| <br /><br />
|-<br />
| Once A file is picked then press “Load” in the Molecule File Browser window.<br />
| <br /><br />
|-<br />
| Going back to VMD Main<br />
| <br /><br />
|-<br />
|<br />
Press “Graphics” and Representations to open a long slender screen.(VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c8cfeb66ca7723b0.png|225x487px]]<br />
|-<br />
| Press the “Create Rep” Button (Graphical Representation window)<br />
| <br /><br />
|-<br />
| With one of the “Reps” highlighted in green, in the drawing Method switch from Lines to VDW<br />
| <br /><br />
|-<br />
|<br />
Change the Sphere scale to 0.2.<br />
<br />
Resolution can stay at default or can be raised if desired.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6ceb4d7e0c7c142a.png|157x340px]]<br />
|-<br />
|<br />
Click on the other line that still says “LINES” and in the Drawing Method change it to Dynamic Bonds<br />
<br />
Change Distance Cutoff to 1.6<br />
<br />
Bond Radius to 0.1<br />
<br />
And Resolution can stay at default or be increased<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_dba6c2b704e3401a.png|158x337px]]<br />
|-<br />
| Go back to VMD Main<br />
| <br /><br />
|-<br />
| Go to “Extensions,” “Visualizations,” and “Movie maker”<br />
| <br /><br />
|-<br />
| Go to the VMD Movie Generator window<br />
| <br /><br />
|-<br />
| In the box of “Name of movie” type the TDESMD_##_### you have.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|214x167px]]<br />
|-<br />
| “Set Working Directory“ choose the destination directory you want the movie to go to.<br />
| <br /><br />
|-<br />
| “Rotation Angle” I put it to 0 but you can put it to whatever.<br />
| <br /><br />
|-<br />
| “trajectory step size” 1<br />
| <br /><br />
|-<br />
| “Movie Duration” This depends on your frames from the (VMDMain) Usually 1000 frames is about 20-30 seconds if the speed (step) is set to 1. But By changing the speed of the video we should make the total video length to 30-45 seconds long.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|232x181px]]<br />
|-<br />
| Back to VMD Main window<br />
| <br /><br />
|-<br />
| Press “Mouse,” “Label,” “Bonds”<br />
| <br /><br />
|-<br />
| click on two neighboring atoms to get the atom name and bond length<br />
| <br /><br />
|-<br />
|<br />
IN VMD Main, go to “Graphics” and “Labels.” select “atoms” in the drop down menu on the “Labels” window.<br />
<br />
Select all atoms by clicking on the top atom and then “SHIFT+left click” the bottom atom to select all atoms<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
While all atoms are selected (highlighted in green). Go to “Properties” and in the “Format” box type %e%i.<br />
<br />
(The %e gives the element and the %i gives the index number)<br />
<br />
Some information can be presented by simply having your cursor over the %e%i and a yellow box will pop up for some info.<br />
<br />
You can move the lettering of each atom by moving the offset information.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_46eab8f828aa40bc.png|215x201px]]<br />
|-<br />
| Click the global properties and change text size and text thickness. ( all of this can be see in the (VMD1.9.3 OpenGL Display) [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_260ebe13a2eda05.png|162x114px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3968014bc41ed6a4.png|224x212px]]<br />
|-<br />
| in VMD movie Generator press “make movie” and quickly in VMD main press the play button.<br />
| <br /><br />
|-<br />
|<br />
Once the video is done a popup should be produced asking where to find videomach. Here is the directory for my computer.<br />
<br />
C:\Program Files (x86)\VideoMach\videomach.exe<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Once you put in the file videomach.exe then videomach should show up.<br />
| <br /><br />
|-<br />
| Switch the file type to MP4<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6a986dcfd4344618.png|374x27px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_da932791910b2a04.png|167x188px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d35497678abed954.png|155x364px]]<br />
|-<br />
| <br /><br />
| Polymer/Monomer Photodegradation: TDESMD 91 121 Bond Data<br />
|-<br />
| <br /><br />
|<br />
Bond lengths shown (white lettering) are chosen to see if this degradation simulation relates to the photo cleavage explained in this specified paper.<br />
<br />
Angew. Chem. Int. Ed. 2015, 54, 1159 –1163<br />
<br />
The transition energy for this excitation is 0.24386664.<br />
<br />
From conformation labeled as &quot;DIMER&quot;<br />
<br />
Acknowledgements: Dr. Dmitri Kilin<br />
|-<br />
| '''Obtaining MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| We will be using python scripts that are premade. In PHOTON<br />
| <br /><br />
|-<br />
| go to the TDESMD_###_### directory you want to go to<br />
| <br /><br />
|-<br />
| in here make another directory called massspectrum<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| mkdir massspectrum<br />
| <br /><br />
|-<br />
| cd masssspectrum<br />
| <br /><br />
|-<br />
| cp /Location/movieALL.xyz<br />
| This should be from the TDESMD_###_### you are in.<br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieALL.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Record number of molecules converted.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| python3 script1n.py<br />
| <br /><br />
|-<br />
| enter number of molecules converted<br />
| <br /><br />
|-<br />
| <br /><br />
| a frequency file should be created<br />
|-<br />
| cp frequency.txt frequency1.txt<br />
| <br /><br />
|-<br />
| awk 'ORS=(!(NR%2)?&quot;\n&quot;:&quot; &quot;)' RS=&quot; &quot; frequency.txt &gt; file1.txt<br />
| <br /><br />
|-<br />
| cp rmovingfile.sh<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
|-<br />
| bash rmovingfile.sh<br />
| <br /><br />
|-<br />
| vi file1.txt<br />
| Edit the file by deleting everything that doesn’t relate to the numbers.<br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| before [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b3731e92878413d2.png|289x328px]]<br />
|<br />
after<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c6aa114dd0a5843f.png|108x333px]]<br />
|-<br />
|<br />
IT IS NECESSARY TO MENTION THAT WHEN CREATING THE FRAGMENTS FILE AN EXTRA BLANK LINE WILL BE GENERATED. IT IS ESSENTIAL TO DELETE.<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_2104ec24e9b14aae.png|258x363px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_36d7bfd4ff6b9e13.png|218x371px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp file1.txt fragments<br />
| <br /><br />
|-<br />
| wc fragments<br />
|<br />
record the first number<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
in my case it is 21<br />
|-<br />
| ~/bin/DRESS_FRAGMENT_v2<br />
| <br /><br />
|-<br />
|<br />
21<br />
<br />
100<br />
<br />
300<br />
<br />
.1<br />
<br />
1<br />
<br />
62<br />
<br />
160<br />
<br />
<br /><br />
|<br />
number of states<br />
<br />
Min MASS (can change if desired)<br />
<br />
Max MASS (can change if desired)<br />
<br />
width of the line used<br />
<br />
magnitude (keep at one)<br />
<br />
number of atoms<br />
<br />
number of electrons<br />
|-<br />
| '''Plotting FERWE MASSPECTRUM'''<br />
| <br /><br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| e in the directory of the file you want to plot, in this example we are using MASS_SPEC.norm as our file with the data we want.<br />
|-<br />
|<br />
set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Monomer H2O Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br />To press the right mouse button and move mouse to preferred window size. Then left click<br />
| monomer h20<br />
|-<br />
| ressing p takes you to the previous zoom level, n to the next level, and u unzooms<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| <br /><br />
|-<br />
| '''Obtaining Multiple Cell MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| Make sure TDESMD CONTCAREXICTED and CONTCARGROUND files have been generated. Start in Cori go to Photon<br />
| <br /><br />
|-<br />
|<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test.sh<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ test_o.sh<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ANA_v2.exe<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/input_ANA<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/CONTCAREXICTED and GROUNDFILES<br />
|<br />
Take these files from Cori and put them into photon.<br />
<br />
These two files are needed for running the scripts properly<br />
<br />
These are the “important parts of the code”<br />
<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
input_ANA should have similar format to this<br />
<br />
Where the first 3 numbers are from POSCAR, the last 3 are the number of cells in the x,y,z axes respectively. This files was duplicated across Z axis.<br />
<br />
18.54438999999999<br />
<br />
14.45937999999995<br />
<br />
17.76793999999994<br />
<br />
1<br />
<br />
1<br />
<br />
10<br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
I used input_ANA<br />
|-<br />
|<br />
CHECK to see if OUTPUT has the correct number of atoms<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_4a92367ce10afd50.png|624x64px]]<br />
| Single cell had 620 atoms, I multipled it by 10 since I wanted to duplicate 10 times. If this is good procede to next steps<br />
|-<br />
| vi test.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_19d33c90b0cefb7a.png|278x215px]]<br />
| change the number to later stage in your trajectory(or the number where you think degradation had occurred. For example my total CONTCARGround and excited was about 191 files. I saw degradation occur later in the simulation so I chose the number 190. This changes with the simulations.<br />
|-<br />
| bash test.sh<br />
| this runs the test.sh file<br />
|-<br />
| bash test_o.sh<br />
| <br /><br />
|-<br />
| NOW LOOKING INTO PHOTON<br />
| <br /><br />
|-<br />
| cp all files into photon directory<br />
| <br /><br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieANA.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
Record number of molecules converted.<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| Follow regular mass spec steps from above. Rather than using MovieALL we are using movieANA.xyz.<br />
| <br /><br />
|-<br />
| <br /><br />
|<br />
photon<br />
<br />
/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
<br />
cori<br />
<br />
/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test<br />
|-<br />
| plummer code for renaming files<br />
| /global/project/projectdirs/m1251/vasp/CHEM676/meade/MONOMER_H2O/TDESMD_162_167/massspec/multiplemassspectest/plumoutput<br />
|-<br />
| '''How to read/get the fragments with their xyz coordinates'''<br />
| <br /><br />
|-<br />
|<br />
Go into the directory with the shot files with the multiple cells. Make sure that some files have this comment inside of them by opening a shot file that has name shot##n#.xyz<br />
<br />
The comment should be.<br />
<br />
created by duplicating cells#<br />
| <br /><br />
|-<br />
| grep -E 'created by duplicating cells.* 17.' shot*<br />
|<br />
finds lines that have both “created by duplicating cells” and the number 17 in the same line.<br />
<br />
17 is the mass number we are looking for<br />
<br />
I am unsure what the “.*” does in this code<br />
<br />
make sure there is a space between the * and the 17<br />
|-<br />
| Then open one of the xyz files listed.<br />
| <br /><br />
|-<br />
| Literally drag and copy the file information into a notepad document in your desktop.<br />
| <br /><br />
|-<br />
| more shot291n10.xyz<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_9c48a58b9e24dc27.png|210x268px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_7c3a89906ed25e30.png|180x270px]]<br />
|-<br />
| open the xyz file in Avogadro and save as mol file.<br />
| <br /><br />
|-<br />
| the Avogadro file is correct visualization of the fragments. Mol file will have some defects so you will have to change those.<br />
| <br /><br />
|-<br />
| Making Video from output of VASP<br />
| <br /><br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exemore<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
I used input_ANA<br />
|-<br />
| '''LAAMPS'''<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_796f3aade1e5b306.png|314x150px]]<br />
|<br />
testwater file:<br />
<br />
Epsolon is Inputs from something like DFT<br />
<br />
<br />
Columns:<br />
<br />
Atom ID<br />
<br />
Molecule ID<br />
<br />
Atom Type<br />
<br />
Q - charge<br />
<br />
X<br />
<br />
Y<br />
<br />
Z<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_f474cba5e65bdaab.png|306x66px]]<br />
| This infor can be put in input script if desired.<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_a9802b534de46367.png|624x180px]]<br />
|<br />
bond id<br />
<br />
bond tuype<br />
<br />
atom id that connects<br />
<br />
atom id that connects as well<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3b00d172c3a15f2b.png|310x103px]]<br />
|<br />
Angle ID<br />
<br />
Angle Type 1 being single bond 2 being double, etc<br />
<br />
Item ID such as first atom O , 2<sup>nd</sup> and 3<sup>rd</sup> atom H<br />
|-<br />
| qsub submit.sh<br />
| submits the job using submit.sh<br />
|-<br />
|<span style="background: #101010">qstat -u meade.erickson</span><br />
| shoes me my jobs<br />
|-<br />
| <span style="background: #101010">chmod -777 lmp_mpi_12Dec18</span><br />
| modifies permission for this file<br />
|-<br />
| [meade.erickson@login0002 testwater]$ chmod -777 ~/bin/lmp_mpi_12Dec18<br />
| giving permission while in a different<br />
|-<br />
| topo readlammpsdata project.data<br />
| reads data file for vmd tk console<br />
|-<br />
| Inputting files, put in .dcd file then right click on it and “Load Data into Molecule” then select .data file.<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_73b2551819922e26.png|287x164px]]<br />
| <br /><br />
|-<br />
| module avail<br />
| Available software in ccast<br />
|-<br />
| qdet job id<br />
| qdel ######<br />
|-<br />
| fragment 0 to 9 or fragment 14<br />
| <br /><br />
|}</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Unix_crash_course&diff=128Unix crash course2022-10-21T17:43:03Z<p>Sysadmin: created page</p>
<hr />
<div>#Introduction to Unix<br />
<br />
<span id="syllabus"></span><br />
=== Syllabus ===<br />
<br />
<ol style="list-style-type: decimal;"><br />
<li>Recommended course material:<br />
<ul><br />
<li>Unix and Linux System Administration 5th Edition - Evi Nemeth, et al.</li><br />
<li>CompTIA Linux+ Study Guide 4th Edition - Chritine Bresnahan and Richard Blum</li></ul><br />
</li><br />
<li>Objectives<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>Shell scripting, input redirection, and C++/C code in a GNU/Linux environment</li><br />
<li>understanding of Linux file system including permissions</li><br />
<li>understanding of command and how to intermingle them to achieve more complex goals</li><br />
<li>basic system architecture, installation process, and management of Linux/Unix</li></ol><br />
</li><br />
<li>General notes<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>Learning to use Bash scripting and <code>vi</code>/<code>neovim</code> essential to professional usage of Linux in real life</li><br />
<li>Any commands discussed in class can be looked up via ‘man’</li></ol><br />
</li></ol><br />
<br />
##History ###What is Unix? 1. Bell System a. Dates back to late 1960s, Originally developed by Ken Thompson and Dennis Ritchie of Bell Labs b. Ritchie described the initial goal as to create a good environment in which to: - do programming and development - foster communal computing through remote access and close communication c. Unix originally written in assembly, but was later re-written in C in 1972 d. first public releases in 1973, at Symposium on Operating Systems Principles; early source code sent by standard mail e. 1975, Version 6 of Unix was licensed to companies for the first time. However, commercial users were rare. Licensees only recieved source code from AT&amp;T. f. Also in 1975, Ken Thompson takes a sabbatical from Bell Labs as a professor at UC Berkeley. Creates BSD: The Berkeley System Distribution of Unix 4.2. - includes C-shell and vi editor for first time g. 1983: Bell broken into several companies in anti-trust suit against AT&amp;T. h. AT&amp;T new restrictive license almost kills commercial Unix. By 1991, BSD had grown to a complete rewrite of AT&amp;T Unix and comes with BSD license. i. BSD survives today as: FreeBSD, OpenBSD, NetBSD, etc. Additionally, Darwin kernel of Mac OSX continues as a form of Unix. 1. GNU Project into Linux a. Around 1983, Richard Stallman creates GNU project b. Stallman’s GNU system goal was to create a ‘free’ system of software that had many of other capabilities of the commercial tools that many Unix also had c. By 1985, Stallman had founded the Free Software Foundation to raise funds to develop free versions of all the components of the Unix syste, under GNU GPL d. In 1991, Linus Torvalds began the development of a new free OS based on the MINIX operation system and SunOS. The 21 year old puts together the kernel. e. As the kernel for the GNU project was still very much in development, the two projects became a natural fit. f. To this day, Stallman insists that Linux is GNU/Linux due to the large reliance on GNU toolchain<br />
<br />
#Introduction to Red Hat Enterprise Linux ###Package Manager - YUM 1. stands for YellowDog Updater Modified 2. provides a set of tools for retrieving software from the internet using subscriptions to an update server 3. utilizes .rpm software to install packages for management 4. creates an Enterprise Linux environment - cons: less software options overall - pro: all available software is tested and works without additional configuration 5. <code>yum</code> has an additional, alternative wrapper program <code>dnf</code> to improve output and ease of use - stands for DaNdified YUM - commands in the format <code>sudo dnf [verb] [objects]</code> - <code>-y</code> flag gives YES as passed option for less attended installation 6. command commands - <code>sudo dnf -y install [package_name]</code> install a particular package and dependencies - <code>sudo dnf -y update</code> update all packages<br />
<br />
#Introduction to Ubuntu ###Package Manager - APT 1. stands for Advanced Package Manager 2. Ubuntu’s set of tools to install packages and keep them up to date 3. utilizes .deb software packages for management 4. keep in mind this package management system is far from standard a. RPM is Linux Standard Base b. Many commerical Unixes have their own package managers c. when all else fails, a tarball (.tar) fille can be used to build from source 5. Commands a. <code>apt-get update</code> synchronizes the index of packages to your system b. <code>apt-get upgrade</code> downloads and installs new versions of all currently installed packages in place, based on the index c. <code>apt-get dist-upgrade</code> downloads and installs new versions, and moves the system up to the next version, based on the index d. <code>sudo</code> was originally “superuserdo”, now works to substitute - users and run programs as that user. /etc/sudoers controls this behavior. e. <code>sudo apt-get install foo</code> : installs package named '''foo''' f. <code>sudo apt-get remove bar</code> : removes a package named '''bar''', but not the config files g. <code>sudo apt-get purge bar</code> : removes a package named '''bar''', and removes config files 8. sudo apt-get source * : gets the source code for a package 9. sudo apt-get clean : removes downloaded packages (.deb files) used to install<br />
<br />
<span id="filesystem"></span><br />
=== Filesystem ===<br />
<br />
<ol style="list-style-type: decimal;"><br />
<li>Directories<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>first and bottom of the directory of the file structure tree is called root '''/'''</li><br />
<li>'''/root''' in addition, there is a file directory, within the root directory called /root (“slash root”), which acts as root’s home directory</li><br />
<li>'''/home''' normal users’ directories</li><br />
<li>'''/boot''' contains the boot directory, including kernel, initial RAMdisk, and bootloader</li><br />
<li>'''/dev''' lists all devices of the computer</li><br />
<li>'''/etc''' configuration files for nearly all programs that need a state</li><br />
<li>'''/tmp''' temporary files for the system</li><br />
<li>'''/var''' variable-length files used by programs<br />
<ul><br />
<li>'''/var/spool''' old-school webmail location</li><br />
<li>'''/var/www''' location of http server files</li></ul><br />
</li><br />
<li>'''/usr''' files available to users, including binaries for daily user tasks in /usr/bin</li><br />
<li>'''/sbin''' binaries used only by root</li><br />
<li>'''/bin''' binaries for the system everyone needs to use</li></ol><br />
</li><br />
<li>Commands<br />
<ol style="list-style-type: lower-alpha;"><br />
<li>'''cd''' change directories</li><br />
<li>'''~''' alias used to shortcut the current user directory</li><br />
<li>'''Ctrl+R''' search history of commands in <code>bash</code>; also available as output from <code>history</code></li><br />
<li>'''!!''' runs the previous command</li><br />
<li>'''ls''' list in the contents of current directory</li><br />
<li>'''.''': the current directory</li><br />
<li>'''..''' : go up one directory#User and Group Configuration</li></ol><br />
</li></ol><br />
<br />
<span id="useraddusermod"></span><br />
=== useradd/usermod ===<br />
<br />
# <code>useradd</code> and <code>usermod</code> do much the same thing<br />
#* <code>-c</code> allows to add comments<br />
#* <code>-d</code> allows to change user home directory<br />
#* <code>-e</code> expiration date - date an account will expire (in YYYYMMDD format), kept in <code>/etc/shadow</code><br />
#* <code>-f</code> specify how many days the account can be inactive before password expires, kept in <code>/etc/shadow</code><br />
#* <code>-g</code> specify default group of the user<br />
#* <code>-G</code> append a list of additional groups for the user<br />
#* <code>-l</code>(usermod only) changes a user login ex: <code>usermod -l darren darseife</code><br />
#* <code>-u</code> uid<br />
#* <code>-s</code> specifies login shell<br />
#* <code>-p</code> specify pre-hashed password<br />
#* <code>-L</code> (usermod only) locks the user’s password so it cannot be entered directly (best used with -e, as well)<br />
#* <code>-U</code> will unlock a user’s password<br />
<br />
###passwd command revisited 1. <code>sudo passwd darseife</code> change the password of darseife 2. <code>passwd</code> can also be used to lock and unlock accounts a. <code>passwd -l</code> will lock the account’s password b. a locked password will always be input incorrectly c. does not necessarily lock user out of the account d. puts an <code>!</code> at the beginning of the password field in <code>/etc/shadow</code> e. <code>crypt</code>, the algorithm used to implement ciphers, cannot generate an <code>!</code>so the password hash is disabled f. <code>passwd -u</code> would unlock the account’s password g. <code>passwd -d</code> removes the password of the account h. <code>passwd -S</code> displays account information: - last password change - password expires - password inactive - account expires - minimum days between password change - maximum days before password change - warn period before changes<br />
<br />
###chage 1. The <code>chage</code> command allows you to modify account settings related to password expiration and aging 2. <code>chage</code> can also be used to configure accounts to automatically expire if: - the password hasn’t been changed - or a date has been passed 3. <code>chage -l</code> print expiration settings of an account 4. <code>chage -m DAYS username</code> to set the minimum time between password changes 5. <code>chage -M DAYS username</code> to set the maximum time between password changes 6. <code>chage -d YYYYMMDD username</code> update the last day YYYY-MM-DD on which to change the password 7. <code>chage -E YYYYMMDD username</code> set the expiration date as YYYY-MM-DD on which username’s account expires - setting the number to <code>-1</code> will make it never expire 8. <code>chage -I DAYS username</code> will lock the account password after number of days inactive 9. <code>chage -W DAYS username</code> number of days the user will be warned before their account is made inactive 10. All of these settings are stored in <code>/etc/shadow</code><br />
<br />
###userdel 1. <code>userdel username</code> deletes users 2. <code>userdel -r username</code> deletes the user, their mail spool, their home directory, and everything in it 3. <code>userdel -f username</code> deletes the user ''right now'' even if they are logged in<br />
<br />
###groupadd and groupmod 1. <code>groupadd</code>add a group to a system 2. <code>groupadd -r</code> used to create a system group, creating a GUID above 1000 3. <code>groupmod</code> change a group 4. <code>groupmod -n</code> change a group name<br />
<br />
###gpasswd - group password 1. <code>gpasswd groupname</code> sets password for a group; it is most common to not have a group password 2. <code>gpasswd -R</code> restricts access to users logging in with password via newgrp<br />
<br />
###getent - get entities 1. can be used to view individual records in <code>/etc/passwd</code>, and other files 2. useful for getting exact matches for items 3. <code>getent password username</code> - get passwd information for user 4. <code>getent shadow username</code> - get shadow information for user 5. <code>getent group groupname</code> - get group information for user#The *nix CLI ###The Shell explained 1. A shell is a command interpreter where the user interacts with the system via command line interface (CLI) 2. Different shells currently in-use A. '''Bourne-Again SHell''' - located at <code>/bin/bash</code> is the main shell and is used was a default on most Linux systems B. '''sh''' - The Bourne Shell, old school version does not require any libraries. Rarely seen as a user-interface shell; typically a symbolic link to another shell binary. Created by Stephen Bourne, first included in Unix V7 C. '''tcsh''' Tee Shell. Based on the earlier C-shell from Unix. Has some additional features built into it. D. '''csh''' C-shell, a BSD component not often seen on Linux. Created by Bill Joy at UC Berkeley. E. '''ksh''' Korn shell from Bell Labs in the 1980s. The idea was to incorporate features from the Bourne Shell along with C-programming syntax. F. '''zsh''' 1990s, incorporates additional ideas into ksh. 3. Built-in Commands A. <code>cd</code> changes directory - <code>cd ~</code> or <code>cd /home/user</code> are equivalent B. <code>ls -</code> lists files in a directory - default behavior just lists files - <code>-a</code> lists all files including hidden ones - <code>-l</code> long format - <code>-d</code> just directories - <code>-F</code> uses a special character to rep filetypes - <code>-R</code> recursive listing C. Single characters are substituted by <code>?</code>, wildcards by <code>*</code> D. Exiting the system - <code>exit</code> exits the current shell - <code>logout</code> only works for login shells E. Shutting down the system - <code>shutdown -h now</code> halts the system now - <code>shutdown -r now</code> restarts the system now - <code>shutdown -h +5 &quot; &quot;</code> broadcasts the message in quotes and tells people it will be down in 5minutes - <code>halt</code> on most systems, will drop the system into single user mode - <code>poweroff</code> “on fire” graceless exit. Sends HW signal to kill power - <code>reboot</code> essentially ‘shutdown -r’ - <code>reboot -f</code> graceless restart exit, Sends HW signal to reboot F. <code>init</code> relies on a series of programs or scripts to change run-state of machine and to start/stop linux services - <code>init 0</code> - shuts the machine down - <code>init 6</code> - reboots the machine - <code>init S</code> - initializes system boot<br />
<br />
###Man pages and getting help 1. displays the manual pages for any packages on the system 2. typing <code>[/]</code> will allow you to search through the page 3. allows you to move via <code>[spacebar]</code> or <code>[pgup/pgdn]</code> 4. <code>man</code> is broken up into seperate chapters: <code>man 5</code> would show chapter 5 1. executable programs and shell 2. system calls 3. library calls 4. device calls 5. file formats 6. games 7. misc. (macro packages) 8. sys admin commands 9. kernel routines 5. <code>man -k *</code> : to find any pages matching <code>*</code> 6. most programs also allow for <code>--help</code> or <code>-?</code> to be passed for a quick rundown on commands and usage<br />
<br />
###Unix filesystem permissions 1. filesystem permissions saved as a series of 3 bit numbers 2. format: <code>d,rwx,rwx,rwx</code> for read-write-execute a. <code>d</code> - denotes a Directory or file b. first triad is for the User of the file c. second triad is for Group members of the file d. third triad is for Everyone else 3. Read-write-execute for files a. '''Read''' allows the file to be opened b. '''Write''' allows for the file to be edited c. '''Execute''' allows for the file to run as a program 4. Read-write-execute for directories a. '''Read''' allows for the directory to view filenames in the directory b. '''Write''' allows files to be added or deleted from a dir c. '''Execute''' allows the directory to be visited, “permission to traverse through”; must have this to execute programs within a directory 5. <code>chmod</code> changes file and directory permissions a. two modes for chmod 1. '''symbolic mode''' <code>chmod {u,g,o}[+,-,=]{r,w,x}</code> 2. absolute mode - chmod 777 /path/to/file` * 0 no permissions * 1 execute * 2 write * 4 read F. chown - change file owner and group 1. chown user /foo - changes foo’s owner to user 2. chown root:root /foo - changes foo’s owner to root, also changes foo’s group to root G. echo - displays any text you enter after the command 1. primarially used in shell scripts or ‘programs’ that utilize a combination of of built-in commands, variables, and executables to complete tasks 2. variables reached from echo by invoking the variable name as $NAME #Input Redirection and Utils<br />
<br />
###Redirection symbols 1. prog &gt; - redirects std output to the file, truncating and rewriting 2. prog 2&gt; - redirect std error to the file, truncating and rewriting 3. &amp;&gt; - redirects both std output and error to file 4. <code>&gt;&gt;</code> - non-destructively appends to end of file rather than rewriting 5. <code>- &quot;backtick&quot; process commands in the middle of another command a. A tool similar to xargs b. text within backticks are treated as a separate command line who's results are substituted on this command line c. things inside of backticks are attempted to be used as a command d. example: rm</code> find / - user darren ` 6. $() - is used like a backtick; more resistant to weird syntax<br />
<br />
###Piping data between programs 1. programs on a UNIX system frequently use pipes to redirect each other’s output. 2. for instance you might use a text-filtering command on output from a file reading command. 3. allows small programs to operate on a flow of information in a row 4. | - this is an unnamed pipe 5. mkfifo - allows for the creation of named pipes a. <code>mkfifo fifo1</code> b. <code>ls -l &gt; fifo1</code> c. <code>cat &lt; fifo1</code> 6. tee - read from standard input and write to standard output and files a. splits standard input so that is it is both displayed via standard output and also captured to one of more files b. allows for logging and redirection while the process is also shown on screen c. overwrites to a file as per ‘&gt;’ by default rather than like ‘&gt;&gt;’ d. example - echo $PATH | tee file1 file2 - file1, file2, and stdoutput all have same information 5. find - searches for files in directory hierarchy 6. xargs - builds a command from standard input, using command name and arguments to that command and then executes the command on each line of standard input a. example: find / - user darren | xargs -d %rm 7. file - returns the type of file and metadata information regarding files 8. join - combines two text files by matching the contents of specified fields a. by default, join uses spaces as the limiters (unless -t CHAR is given) b. example: join -1 3 -2 2 - this joins the third field of the first file and the second field of the second file 9. paste A. Paste is like cat rotated 90 degrees. When operating on multiple files, it will merge the lines of the files. 10. expand - converts tabs to spaces A. by default assumes you want 8 spaces per tab; -t changes B. unexpand - converts spaces to tabs 11. od - dump files into octal and other formats A. default usage is dumping binary file into octal binary format B. -x for 2byte hex output 12. sort - sorts lines of text files A. writes sorted concatenation of all files as output B. by default, uses first column C. Important options 1. -f : ignore case 2. -m : sort by three digit month abbreviation; ex, jan 3. -n : numeric sort 4. -v : sort by versio 5. -h : compare human readable numbers 6. -r : reverses the sort 7. -R : sorts randomly but groups like files 8. -d : dictionary-order 9. -k : sort via key, sorts by a field number, can be comma-seperated listq D. the opposite of this is ‘shuf’ 13. split - splits a file into 2 or more files A. needs to have an output file prefix B. where do you want to split 1. -b : size in bytes 2. -l : lines C. example: split -l 2 listening1.txt numbers -&gt; output is: ‘numbersaa’ and ‘numbersab’ D. by default: splits every 10,000 lines 14. tr - trnaslate or delete characters A. changes individual characters coming in through standard input and outputs via standard output B. options 1. [:lower:] - all lower case chars 2. [:upper:] - all upper case chars 3. [::] 15. cut - extracts portions of lines of input and prints out as output A. used to remove items within lines, delimited by spaces or tabs B. options 1. -b : cuts the specified list by byte 2. -c : cuts the specified characters 3. -f : cuts specified list of fields from input 4. -d : specifies the delimiter for fields C. EXAMPLE: cut -f 7 -d : /etc/passwd 16. uniq - take an input list and report or omit repeated lines A. only if they are next to each other, so you have to sort first B. EXAMPLE: cut -f 7 -d : /etc/passwd | sort | uniq<br />
<br />
<span id="c-programs-and-makefiles"></span><br />
=== C++ programs and Makefiles ===<br />
<br />
# Basic C++ program with input and output<br />
<br />
<pre>int main (int args, char * argv[]){<br />
for(int x = 0; x &lt; argv[],x++){<br />
cout &lt;&lt; arv[x] &lt;&lt; endl;<br />
}<br />
return 0;<br />
}<br />
// outputs every character given to it</pre><br />
<ol start="2" style="list-style-type: decimal;"><br />
<li>Let’s write a less good version of grep called greb:</li></ol><br />
<br />
<pre>#include &lt;iostream&gt;<br />
#include &lt;fstream&gt;<br />
using namespace std;<br />
<br />
int main(int argc, char * argv[]) {<br />
string curLine;<br />
if (argc == 2){<br />
while(getline(cin,curLine)) {<br />
if (curLine.find(argv[1]) != string::npos) {<br />
cout &lt;&lt; curLine &lt;&lt; endl;<br />
}<br />
}<br />
}<br />
<br />
else if (argc == 3){<br />
ifstream inFile(argv[2], ifstream::in);<br />
<br />
while(getline(inFile,curLine)) {<br />
if (curLine.find(argv[1]) != string::npos) {<br />
cout &lt;&lt; curLine &lt;&lt; endl;<br />
}<br />
}<br />
}<br />
else {<br />
cerr &lt;&lt; &quot;Error: unknown parameters.&quot; &lt;&lt; endl;<br />
return 1;<br />
}<br />
return 0;<br />
}<br />
</pre><br />
<ol start="3" style="list-style-type: decimal;"><br />
<li>Make A. Allows for automatic configuration and creation of usable binararies from OOP files, and can be used to install and uninstall programs B. Allows for the creation of archives of source code that build as packages into useful code C. typically named ‘makefile’</li></ol><br />
<br />
<pre># set a variables for the compiler<br />
CC = g++<br />
LC_MESSAGES=C<br />
CFLAGS=-g -Wall<br />
<br />
# target name variable<br />
TARGET = greb<br />
<br />
# file extension<br />
FILEXT = cpp<br />
<br />
# the build target executable =&gt; install path of the executable<br />
PREFIX = /usr/local<br />
DESTDIR = /home/bernd/staging/<br />
<br />
# basic makefile format<br />
# name of instruction: dependencies<br />
# &lt;tab&gt; command to execute<br />
# the FIRST one you list is the default<br />
$(TARGET): $(TARGET).$(FILEXT)<br />
$(CC) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)<br />
<br />
.PHONY: all<br />
all: $(TARGET).$(FILEXT)<br />
$(CC) $(MAKEOPTS) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)<br />
<br />
.PHONY: clean<br />
clean:<br />
rm $(TARGET)<br />
<br />
.PHONY: install<br />
install: $(TARGET)<br />
mkdir -p $(DESTDIR)$(PREFIX)/bin<br />
cp $&lt; $(DESTDIR)$(PREFIX)/bin/$&lt;</pre><br />
<span id="tar-and-packaging"></span><br />
=== Tar and Packaging ===<br />
<br />
# tar A. the name ‘tar’ comes from its original purpose: Tape ARchive B. you can bundle files as .tar, ‘tarballs’, which contain a series of file entries and terminating bytes between files C. each entry has a file descriptor and the binary contents of the file D. file descriptor, or header, contains the name of a file, a checksum of that file, and permissions data E. does not compress data, but instead keeps it in one file F. tar options: -c: Creates a tar archive -v: Verbose output -x: eXtract from a tar archive -f: specify a Filename -A: Appends two archives together -M: create or extract a multivolume archive -j: specify bzip2 compression -J: specify xz compression -z: specify gzip compression –lzma: specify lzma compression -t: Test an archive<br />
<br />
<span id="more-unix-commands"></span><br />
=== More Unix Commands ===<br />
<br />
# fmt - format A. some files have crazy long lines of text B. reformats to a certain -width C. by default, truncates after 75 characters and cleans up paragraphs D. anything with 2+ blankline removed, and adds indention E. -s : prevents truncation of paragraphs<br />
# nl - line number A. nl does count whitespace by default B. identical to cat -b (but not cat -n)<br />
# pr - prepare a file for printing A. includes header, footer, and page breaks B. create colummns: pr -3 file.txt C. -l sets the length of lines D. -o choose the header text<br />
# regular expressions - sed and awk A. simliar to wildcard expansion: ex; ls ''.txt B. grep and sed are two examples of regEx programs C. unix system programs have two different forms of regular expressions: basic and extended D. types of expresssions: 1. exact text match: grep “blah” returns all lines with “blah” 2. bracketed expressions: b[aeiou]g - returns bag, beg, big, bog, bug 3. bracketed with range: a[2-4]z - returns a2z, a3z, a4z 4. matching single char: a.z (allows for anything between a and z) 5. matching begin/end of line: ^ beginning, $ end 6. any of these can be combined with the wildcard: '' ex: A.* Lincoln (with 0 or more occurances of a pattern.) ex: A[ae]* Lincoln (with 0 or more occurances of a-e after A. 7. matching at least one of: + symbol ex. A[ae]+ Lincoln - returns at least Aa or Ae Lincoln 8. multiple possibilities or: | needs to be set up with ’ ex. ‘Darren (F|Fredrick) Seifert’ 9. grep -E invokes regex functionality<br />
<br />
<span id="shell-scripting"></span><br />
=== Shell Scripting ===<br />
<br />
# The first line of a shell script tells which program to use to interpret the contents: #!/bin/bash or another shell A. the first two characters are (#!) are special code to tell the kernel this is a script - crunchbang, shebang, hashbang, poundbang B. the remaining lines beginning with # are considered comments <code># this is a comment</code><br />
# Classic Hello World<br />
<br />
<pre># hello.sh<br />
#!/bin/bash<br />
# Print out hello world<br />
echo 'Hello World!'</pre><br />
<ol start="3" style="list-style-type: decimal;"><br />
<li>When you want to run a shell script, it needs to be made executable before it is an executable file: ‘# chmod u+x ./file.sh’</li><br />
<li>Can be invoked with ./file.sh</li><br />
<li>Alternative routes: A. bash file.sh works whether or not #!/bin/bash or chmod +x has been given B. exec file.sh is used to run scripts from within other scripts and to stop the tty instance after it is done (used to run scripts within scripts)</li><br />
<li>Passing Parameters A. much like C++ programs, bash scripts can be passed parameters B. accessing these, the dollar sign is used, much like PHP or Perl, to call the number of the parameter 1. $0 - the name of the running script itself 2. $1 - the first parameter 3. and so on… 4. $? - returns the exit status of the last run command 5. $$ - returns the process ID (PID) of the current script 6. $USER - returns the username of the user running the script 7. $SECONDS - returns the runtime of the current script 8. $RANDOM - generates a random number (int 0 - 32000) 9. $LINENO - current line of script C. creation of new variables 1. variablename=value # no spaces ever 2. if you do want spaces, they must be escaped by quotes 3. double quotes recommended for variable substitution later 4. for instance varname=“value $1” allows for insertion of other variables into variables whereas varname=’value <math display="inline">1' will not work D. saving output into variables 1. output from programs can be saved into a variable 2. example: curdir=</math>(pwd) will run pwd and return the standard output as a variable 3. if you want to capture the standard error for a command, it can be saved as <code>error=$(&lt;/tmp/error)</code> 4. alternative use: curdir=<code>pwd</code> note the backtics</li></ol><br />
<br />
<pre># foo.sh<br />
#!/bin/bash<br />
# An example script<br />
cp $1 $2<br />
#verify it worked<br />
echo &quot;Details for $2&quot;<br />
ls -l $2</pre><br />
* handling user input and output can use ‘echo’ to create interactive scripts:<br />
<br />
<pre># plususer.sh<br />
#!/bin/bash<br />
echo -n 'Please enter user name for account:'<br />
read username # this saves the stdin input as username<br />
read -p 'Username' $username # -n no-newline<br />
echo $username<br />
read -sp 'Password:' password # -s stealth</pre><br />
* scripts can also handle files as standard input like other commands<br />
<br />
<pre># salesreport.sh<br />
#!/bin/bash<br />
#a sales report generated from standard input<br />
echo Data Summary:<br />
cat /dev/stdin | cut -d ' ' -F2,3 | sort</pre><br />
* variables can be made global via export<br />
<br />
<pre># script.sh<br />
#!/bin/bash<br />
var1=blah<br />
var2=foo<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
export var1<br />
exec ./script2<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre><br />
<pre>script2.sh<br />
#!/bin/bash<br />
echo $0 :: var1 : $var1 , var2 : $var2<br />
var1=flop<br />
var2=bleh<br />
echo $0 :: var1 : $var1 , var2 : $var2</pre><br />
<span id="sed---stream-editor"></span><br />
=== Sed - Stream EDitor ===<br />
<br />
# the sed command uses regular expressions to modify stdin and sends the changes to stdout<br />
# usage : <code>sed [options] script.txt [inputfile]</code><br />
# some of the commands that can be used in the script portion<br />
#* <code>=</code> - display the current line number<br />
#* <code>a\text</code> - append text to file<br />
#* <code>i\text</code> - insert text to file<br />
#* <code>r file</code> - append text from filename into the file<br />
#* <code>c\text</code> - replace the selected range with the text<br />
#* <code>s/2012/2013/</code> - replace the first occurrence of 2012 on first occurrence on each line<br />
#* <code>s/2012/2013/g</code> - replace ’’ ’’ every occurrence on each line<br />
# examples<br />
#* <code>sed 's/Nick\|nick/Gary/g' &lt; salaryprojections.txt</code><br />
#* ^ this would replace any ‘Nick’ or ‘nick’ with ‘Gary’ at all points in the file<br />
#* standard replace format: <code>sed 's/^/ /' file.txt &gt; file.new.txt</code> ### Examining Processes<br />
# <code>ps</code> command<br />
#* The internal Unix command for process control is <code>ps</code><br />
#* By default, <code>ps</code> reports back what is happening inside of the current terminal/tty;<br />
#* shows <code>PID</code>, <code>TTY</code>, <code>TIME</code>, and <code>CMD</code><br />
#** <code>PID</code> is the Process ID number for a running program<br />
#** <code>TTY</code> is the Teletypewriter device which the process runs from; reports stdout and stderr to this by default<br />
#** TIME is the cumulative processor time used by the program on Linux<br />
#* <code>ps</code> supports traditional Unix options and GNU flags, which can be somewhat confusing<br />
#* Options for ps<br />
#** Display all processes running <code>ps -A/-e</code> both cause all processes on the system to be listed and output is identical<br />
#** Display processes belonging to a specific user: <code>ps -u user</code><br />
#** Display extra fields as a “full format” <code>ps -f</code> can be combined with other options<br />
#*** <code>UID</code> User ID shows the username of the user running the command<br />
#*** <code>PPID</code> parent process ID, the process ID of the process that spawned the process<br />
#*** <code>C</code> processor utilization as % of usage over the lifetime of the process<br />
#*** <code>STIME</code> start time of the process as per system time (only in 24h clock format if current day; otherwise uses MMDD format)<br />
#** Display long format <code>ps -l</code><br />
#*** <code>F</code> Flags field<br />
#**** <code>0</code> = no special flags for this process<br />
#**** <code>1</code> = forked process, but did not call exec<br />
#**** <code>4</code> = executable was executed with superuser privileges<br />
#*** <code>S</code> State codes<br />
#**** <code>D</code> Uninterruptible sleep (usually IO)<br />
#**** <code>R</code> Running or runnable (on run queue)<br />
#**** <code>S</code> interruptible Sleep (waiting for an event to complete)<br />
#**** <code>T</code> Stopped, either by a job control signal or because it is being traced.<br />
#**** <code>X</code> dead (should never be seen)<br />
#**** <code>Z</code> Defunct (“zombie”) process, terminated but not reaped by its parent.<br />
#*** <code>UID</code> uses system number rather than human readable name<br />
#*** <code>PRI</code> Priority is used by the kernel, rather than in userspace, to determine the next in queue for processing<br />
#*** <code>NI</code> Niceness of a process describes how accommodating a process is to sharing time on the system.<br />
#**** This number ranges from +19 to -20, with 0 as the default.<br />
#**** In the scheduling system, niceness is used to determine how much processing to give to each process when it asks for it.<br />
#*** <code>ADDR</code> the memory address of the process<br />
#*** <code>SZ</code> the size of the image in pages<br />
#**** how much virtual memory is in use by the process.<br />
#**** Includes text, data, and stack space<br />
#**** page size is a kernel variable and can be read by <code>getconf PAGE_SIZE</code><br />
#*** <code>WCHAN</code> address of where a function is sleeping (only kernel functions)<br />
#*** <code>RSS</code> amount of physical memory in use<br />
#* User-oriented format <code>ps -u</code> displays information in a more readable manner<br />
#** <code>%CPU</code> percent of processor time used by process divided by real time<br />
#** <code>%MEM</code> actual percentage of non-reserved physical RAM<br />
#** <code>VSIZE</code> virtual memory size used<br />
#* Virtual memory format <code>ps -v</code> displays information including number of:<br />
#** <code>MAJFL</code> major page faults of the process, indicating times it has used swap space<br />
#** <code>TRS</code> physical memory devoted to stack<br />
#** <code>DRS</code> physical memory devoted to non-executable code<br />
#* Recieve a process hierarchy <code>ps -AH</code> see parent-child relationships visually<br />
#* Choose your own adventure mode <code>ps -o uid,pid,ppid,command</code><br />
#** this command used to determine certain columns<br />
#* Piping ps can be piped as per normal to other programs:<br />
#** Ex; <code>ps -u darseife \| grep bash</code><br />
#** <code>pgrep</code> is a common utility that behaves similarly to the previous command<br />
# <code>top</code><br />
#* <code>top</code> is used to view processes as they execute in real time<br />
#** <code>-d</code> used to choose update frequency in seconds<br />
#** <code>-p</code> pid to monitor specifically<br />
#** <code>-n</code> run for N iterations<br />
#** <code>-r</code> command to renice a process within top<br />
#** <code>-s</code> to change the update rate within top<br />
#* <code>htop</code> can also be used as a next generation to top<br />
# <code>nice</code> and <code>renice</code><br />
#* The <code>nice</code> command is used to run a command with an adjusted niceness<br />
#** generally can’t give negative values to programs without superuser access<br />
#** <code>nice -n 12 ./script.sh</code> will add 12 to the default priority (0)<br />
#* <code>renice</code> is used to adjust niceness of running processes.<br />
#** <code>renice 0 -p 2772 3562</code> renices the processes 2772 and 3562 to 0<br />
#** <code>renice 1 -u darseife</code> renices the processes of user darsiefe to 1<br />
# Killing processes<br />
## <code>kill</code><br />
##* stopping a process currently running <code>-kill -s signal PID</code><br />
##** <code>PID</code> PID by default, kill sends a signal to a number of process IDs<br />
##** signal 1 is <code>SIGHUP</code> terminates interactive programs and causes daemons to reread config files<br />
##** signal 9 is <code>SIGKILL</code> stops all processes without performing a graceful exit or shutdown<br />
##** signal 15 is <code>SIGTERM</code> asks the process to terminate and to close out open files, run its shutdown script, etc.<br />
##* <code>kill</code> will only kill processes owned by the user, unless run with superuser access<br />
## <code>killall</code><br />
##* kill multiple things<br />
##* <code>killall vi</code> kills all running instances of vi for the user; when run as a superuser, this will kill any running instance of vi<br />
##* <code>killall -s 9 vi</code> kills all running instances of vi for the user by sending the <code>SIGKILL</code> signal to it; when run as a superuser, this will kill any running instance of vi.<br />
##* <code>killall -i vi</code> uses interactive mode to ask which processes to kill<br />
##* if you are running killall as root, try to use -i to prevent loss of other people’s work<br />
# Additional process control<br />
#* If you are running a program inside of a terminal, <code>[Ctrl + Z]</code> will pause and go to background<br />
#** To restart the process, run the command <code>fg</code> in the same terminal<br />
#** With multiple processes paused, <code>fg 1</code> or <code>fg 2</code> will bring back process 1 or 2<br />
#* Processes can be started in the background with &amp; at the end of the line; unlike paused processes, these will continue to run.<br />
<br />
<span id="linux-time-and-scheduling"></span><br />
=== Linux Time and Scheduling ===<br />
<br />
# Overview<br />
#* Computer systems in general rely on a pair of clocks to operate<br />
#* Two clocks<br />
#** '''Hardware clock''' - initialized in BIOS/UEFI and kept updated via battery while computer is off<br />
#** '''Software clock''' - initialized at boot from BIOS/UEFI time, and is used to track time while running<br />
#* Typically, the clock of the computer is set using coordinated universal time (UTC)<br />
#** adjusted for use based on timezones and localization settings like DST before entering userspace<br />
#* Both the hardware clock and the software clock are not terribly reliable and tends to drift over time<br />
#* This can be a problem on x86_64 systems in general because many network services rely on accurate timekeeping<br />
#* This is made worse when having hundreds or thousands of servers requesting time from a single source could be bad<br />
# Unix Time Commands<br />
#* <code>date</code> outputs the current software clock time<br />
#* <code>date -u</code> outputs the UTC time<br />
#* <code>date 102715022019</code> sets the time to exactly Oct 27, 15:02, 2019 in local time<br />
#* <code>hwclock</code> used to retrieve the value of the hardware clock; must be done as root<br />
#* setting the hardware clock<br />
#** <code>hwclock --set--date 102715022019</code> is a manual set<br />
#** <code>hwclock -hctosys</code> set the hardware clock to the system clock<br />
#** <code>hwclock --synctohc</code> synchronizes system clock to hardware<br />
#* these are called by daemons and init on startup and shutdown<br />
#* both the system time and the hardware clock still suck and need a replacement<br />
# <code>NTP</code> Network Time Protocol<br />
#* NTP relies on a tree of servers (each level is called a stratum) from a hardware atomic clock server<br />
#* The further down the tree you are the less accurate the time is.<br />
#* One NTP server requesting time from a center will exchange packet via timestamp and the delay between communications is estimated<br />
#* It is best to pick NTP servers with a short delay<br />
#* Typically, a cluster of computer is requested from a local time server, which in turn is the only computer requesting times outside of the network<br />
# <code>timesyncd</code><br />
#* Traditionally, <code>ntpd</code> is the daemon that has provided client time synchronization<br />
#* Newer versions of Ubuntu use a service called <code>timesyncd</code><br />
#** uses a slightly different protocol than NTP, called <code>SNTP</code> (simplified network time protocol)<br />
#** configuration information is at <code>/etc/systemd/timesyncd.conf</code><br />
#** <code>timedatectl</code> is a way to view the current SNTP settings<br />
#* also more information related to this accessible with <code>sudo systemctl status systemd-timesyncd</code><br />
# Running jobs in the future: <code>cron</code><br />
#* <code>cron</code> like timesyncd or ntpd, cron is a daemon that runs continuously and looks for events to cause it to spring into action<br />
#* Most daemons are network services, but cron responds to temporal events<br />
#* A cron daemon wakes up once per minute, and examines its config files in:<br />
#** <code>/var/spool/cron/</code><br />
#** <code>/etc/cron.d/</code><br />
#** <code>/etc/crontab</code><br />
#* Executes command specified:<br />
#** System jobs - run as root, and perform system maintenance tasks (rotate logs, clean out <code>/tmp</code>, remove old files, update system)<br />
#** User jobs - run jobs in the future for users<br />
#* <code>/etc/crontab</code><br />
#** System cronjobs are controlled by this file<br />
#** Normally, begins with environment variable configuration, then several lines like:<br />
#*** On a typical system <code>02 4 * * * root run-parts /etc/cron.daily</code><br />
#*** On Ubuntu Desktop <code>25 6 * * * root test -x /usr/sbin/anacron || cd / &amp;&amp; run-parts --report /etc/cron.daily</code><br />
#** Fields for cron, in order<br />
#**# the minute of the hour<br />
#**# the hour of the day<br />
#**# day of the month<br />
#**# month of the year<br />
#**# day of the week<br />
#** different scripts for items are run from:<br />
#*** <code>/etc/cron.daily</code><br />
#*** <code>/etc/cron.weekly</code><br />
#*** <code>/etc/cron.monthly</code><br />
# <code>anacron</code> a specific implementation of cron<br />
#* <code>anacron</code> is not intended to be a replacement for cron, it is a supplement<br />
#* cron works great on systems that are ''always on''<br />
#* <code>anacron</code> is smart in that it can tell if things are run<br />
#** if things are missed, it plays catchup<br />
#** light-night log rotation, for example, might never run<br />
#** anacron is intended to address this<br />
#*** when run it checks to see when it last executed each program<br />
#*** if a period greater than the programs interval has passed, the program gets run by anacron<br />
#** config file <code>/etc/anacrontab</code><br />
#*** format: <code>1 5 cron.daily run-parts --report /etc/cron.daily</code><br />
#**** first column: delay in minutes<br />
#**** second column: frequency in days<br />
# <code>crontab</code> edit cronjobs for users<br />
#* cron jobs for the system are under <code>/etc/</code> but user ones are kept in <code>/var</code><br />
#* <code>crontab -e</code> edit your user cronfile<br />
#* root can edit other users with <code>crontab -u username -e</code><br />
#* <code>/etc/cron.allow</code> and <code>/etc/cron.deny</code> exist for whitelisting or blacklisting users from using <code>cron</code><br />
#** an empty <code>/etc/cron.deny</code> allows ALL USERS if present<br />
#** an empty <code>/etc/cron.allow</code> blocks ALL USERS if present<br />
#* if neither of these exist, then only root may use <code>cron</code><br />
# <code>at</code> a handy simple task timer<br />
#* cron and anacron can be overkill for simple tasks that need to be done once<br />
#* example:<br />
#** <code>at 5am Oct 31</code><br />
#** <code>at&gt; /home/darren/run-backup.sh</code><br />
#** <code>at&gt; echo &quot;job done&quot;</code><br />
#** <code>at&gt; mail -s &quot;backup job finished&quot; root</code><br />
#** <code>at&gt;</code> '''ctrl+d''' to finish<br />
#* <code>at now +1 hour</code> will run 1 hour from now relative to current system time<br />
#* other examples:<br />
#*# <code>at midnight</code><br />
#*# <code>at noon</code><br />
#*# <code>at teatime</code><br />
#* <code>aterm</code> to remove an upcoming command<br />
#* <code>atq</code> used to list pending commands<br />
<br />
<span id="log-files"></span><br />
=== Log Files ===<br />
<br />
# Intro<br />
#* Linux machines log files that record key details about system operation and are typically stored in <code>/var/log</code><br />
#* Logging on Linux system is generally performed by the syslog daemon, <code>syslogd</code><br />
#** However, some daemons and programs perform logging independently of syslog<br />
#* Logging on Linux can be done locally to the computer, but in a networked environment, logs may be sent to a central logging server and by email<br />
#* Logging is a security action and is typically a part of the system worth hardening because it is exploitable by an attacker to hide their break-in<br />
#* Logs should be rotated, aka moved out of the system, to prevent <code>/var/log</code> from being filled completely<br />
# <code>syslog</code><br />
#* The basic idea of a system logger is to provide a unified means of a handling log files<br />
#* The daemon runs in the background and and accepts data delivered from servers and other programs<br />
#* Once the log daemon receives a message, it is classified and directed to an appropriate file<br />
#* <code>syslogd</code> might be managing several services at the same time and providing different rule-based logging based on priority of the messages received<br />
#* <code>syslogd</code> relies on the config file <code>/etc/syslog.conf</code><br />
#** Ubuntu uses <code>/etc/rsyslog.conf</code> for rocket-fast syslog<br />
#** the configuration file supports a ruleset, located at <code>/etc/rsyslog.d/</code> or <code>/etc/syslog.d/</code> respectively<br />
#** standard logging is done by daemon, with the format <code>facility.priority action</code><br />
#*** example: <code>mail.* /var/log/mail.log</code><br />
#** facility parameter is the name of the type of program the daemon that generates messages to be logged<br />
#**# <code>auth/priv</code> security messages<br />
#**# <code>cron</code> from the cron daemon<br />
#**# <code>daemon</code> catch-all for various systems servers<br />
#**# <code>kern</code> kernel messages<br />
#**# <code>lpr</code> printer and CUPS messages<br />
#**# <code>mail</code> mail subsystems<br />
#**# <code>news</code> oldschool newsgroups<br />
#**# <code>syslog</code> internally generated messages<br />
#**# <code>uucp</code> unix-to-unix copy daemon<br />
#**# <code>local[0-7]</code> predefined facilities for custom configurations<br />
#** priority parameters include:<br />
#*** <code>deb</code> debugging messages<br />
#*** <code>info</code> normal operations<br />
#*** <code>notice</code> abnormalities that do not effect functionality<br />
#*** <code>warn</code> tells that an error will occur is action is not taken<br />
#*** <code>err</code> standard error for the daemon, non-urgent failure of the daemon causing an error message<br />
#*** <code>crit</code> critical error causing a secondary system to fail urgently<br />
#*** <code>alert</code> critical error causing a primary system to fail urgently<br />
#*** <code>emerg</code> Panic message - system unusable<br />
#** the action parameter is a file or location on the network that will accept the message<br />
#*** could be a file in <code>/var/log</code><br />
#*** could also be a tty: <code>/dev/tty8</code><br />
#*** could also be a computer on the network: <code>@logger.ndus.edu</code><br />
#* When a program sends a message to the system logger, it includes the priority: the message is only logged if it is of a certain level or higher<br />
#** Exceptions:<br />
#*** <code>*</code> all codes<br />
#*** = <code>(mail.=crit)</code> applies to this priority only<br />
#** You can also specify multiple facilities with the same priority by using a comma<br />
#** Multiple facilities separated by commas followed by a priority all follow the same priority<br />
#** A semicolon is used to specify a number of logging priorities <code>kern.info;kern.err</code> everything between info and error<br />
# <code>logger</code> - logging something else<br />
#* The <code>logger</code> command allows a user to log something else other than established facilities<br />
#* <code>logger shutting down for system maintenance</code><br />
#** will log to <code>/var/log/syslog</code> as <code>Nov 6 15:50:00 darseife-virtualbox darseife: shutting down for system maintenance</code><br />
#* <code>-i</code> records the PID of the logger process when run (useful when run from script)<br />
#* <code>-s</code> print the log message to standard error in addition to logging<br />
#* <code>-f /foo</code> logs to <code>/foo</code> instead of the system log<br />
#* <code>-p</code> specify both a facility and a priority for the system log message; <code>-p local3.info</code><br />
<br />
<span id="firewalling"></span><br />
=== Firewalling ===<br />
<br />
# TCP Wrappers<br />
#* TCP wrappers depend on two config files to work properly; these are <code>/etc/hosts.allow</code> and <code>/etc/hosts.deny</code><br />
#* the most common config for these is default <code>deny</code><br />
#** <code>ALL:ALL</code> in <code>/etc/hosts.deny</code><br />
#** <code>ALL:127.0 [::1]</code> in <code>/etc/hosts.allow</code><br />
#** specific services from a specific host enabled as the form <code>sshd:192.168.X.Y</code> in <code>/etc/hosts.allow</code><br />
#** all services from a domain as <code>ALL:.minotstateu.edu</code> in <code>/etc/hosts.allow</code><br />
#* Also used is specific allow<br />
#** nothing is listed in <code>/etc/hosts.allow</code><br />
#** specifically removed IP addresses in the form <code>ALL:165.234.X.Y</code> in <code>/etc/hosts.deny</code><br />
#* Secondary service-level blocking can be used to apply rules to services in the form <code>/etc/sshd.hosts</code><br />
# <code>iptables</code><br />
#* <code>iptables</code> is the kernel level netfilter firewall<br />
#* <code>iptables -L</code> lists the current ruleset<br />
#* <code>iptables -A</code> is used to APPEND new rules to a chain (takes a chain name)<br />
#* <code>iptables -I</code> inserts a rule (takes a chain name and a position)<br />
#** example <code>iptables -A INPUT -ptcp -dport ssh -j ACCEPT</code><br />
#* rule options<br />
#** <code>-p</code> the protocol used<br />
#** <code>-d</code> the destination port, can accept a range of port<br />
#** <code>-j</code> the action to take, includes ACCEPT, DENY, DROP<br />
#** <code>-s</code> specify a source address, can be either <code>192.168.0.1</code> or <code>192.168.0.0/24</code><br />
#** <code>-m</code> require the rule to match a specific property<br />
#*** ex <code>iptables -m conntrack</code> allows filter rule to match based on connection state<br />
#** <code>--ctstate</code> define a list of connection states for the rule to match on<br />
#**# NEW - new connections<br />
#**# RELATED - new but related to a previous on<br />
#**# ESTABLISHED - a preexisting connection already in progress<br />
#**# INVALID - any malformed traffic or unidentified traffic<br />
#** <code>-j</code> tells iptable what to do with the packet given the rule<br />
#*** ACCEPT - routes as normal as per IP header<br />
#*** DROP - kills the packet without giving reply<br />
#*** REJECT - sends an explicit error message<br />
# Uncomplicated Firewall (ufw)<br />
#* <code>ufw enable</code> to turn on ufw<br />
#* <code>ufw disable</code> to turn off ufw<br />
#* <code>ufw status</code> view current rules for ufw<br />
#* <code>ufw status verbose</code> reports logging level + access levels for each rule<br />
#* <code>ufw allow 80/tcp</code> sets a rule to allow tcp packets on port 80<br />
#* <code>ufw deny 22/udp</code> sets a rule to reject udp packets on port 22<br />
#* <code>ufw deny ssh</code> sets a rule to reject all ssh protocol packets<br />
#* <code>ufw allow 6000-6007/tcp</code> sets a rule with a range to allow tcp packets from 6000-6007<br />
#* <code>ufw allow from 15.15.15.1 to any</code> sets a rule allowing all traffic from 15.15.15.1<br />
#* <code>ufw status numbered</code> returns a numbered list of rules in the set<br />
#* <code>ufw delete 2</code> removes rule number 2 in a list<br />
#* <code>ufw logging {on/off}</code> turns the logging on or off; usually caught by syslogd/rsyslog; logging is set to <code>low</code> by default<br />
#* <code>ufw logging {low/medium/high/full</code> sets logging verbosity of ufw;<br />
#** note that logging is rate limited at <code>medium</code> and not rate limited at <code>high</code><br />
<br />
<span id="bootinitialization"></span><br />
=== Boot/Initialization ===<br />
<br />
# Slackware - SysV Init<br />
#* System 5 style rc script via <code>init</code><br />
#* <code>PID 1</code> = <code>init</code> which is the process that starts all other processes when the system boots<br />
#* <code>init</code> is responsible for processing <code>/etc/inittab</code> to see how to start the system and at what runlevel<br />
#* Things <code>init</code> does<br />
#** Runs the script <code>/etc/rc.d/rc.s</code> to determine to prepare the system before loading to the desired runlevel<br />
#** Enables virtual memory for the system and mounts filesystems in <code>/etc/fstab</code><br />
#** Loads kernel modules<br />
#** Near the end of the script <code>rc.s</code>, init will run scripts related to the runlevel the system is entering -for example, <code>rc3.d</code> is run at runlevel 3<br />
# SystemD<br />
#* Ubuntu, CentOS, RedHat, and others have moved from SysV init process to the SystemD initialization process<br />
#* Rather than running many shell scripts during init, the systemD process uses one monolithic program to control startup of services<br />
#* The way the services are started are controlled by a configuration file<br />
#* systemd uses ''units'' and ''targets'' rather than scripts and runlevels<br />
#** there are 11 different types of units:<br />
#*** <code>mount</code> unit defines a mountpoint on the system named with the mount path with slashes replaced as dashes<br />
#*** <code>swapspace</code><br />
#*** <code>device</code> a device in the linux device tree<br />
#*** <code>socket</code> fifo pipe to an internet port with an associated service<br />
#*** <code>service</code> a service or application on the system<br />
#** targets are used to stop and start the service and group units together<br />
#** systemd identifies units by name and type<br />
#*** example ssh could have an ssh.service and an ssh.socket<br />
#* <code>systemctl</code> - can list current units loaded on the system<br />
#* config files - <code>/lib/systemd/system</code><br />
#* modify how a unit functions - <code>/etc/systemd/system</code></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Main_Page&diff=127Main Page2022-10-19T20:29:52Z<p>Sysadmin: </p>
<hr />
<div><div><br />
=Welcome to the Rasulev Research Group Wiki!=<br />
==The Computational Polymer Science and Cheminformatics Group at North Dakota State University==<br />
<br />
This page hosts the tutorials and procedures for the group. Implementing MediaWiki, it provides a platform to organize knowledge and make it available to both lab collaborators and students. <br />
<div style="clear: both;"></div><br />
<div id="audiences" class="mainpage_row"><br />
<div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon chem.svg|20px|middle|link=]]</span><span>Lab Procedures and Software Tutorials</span></h3><br />
<div> <br />
* [[ADF ReaxFF]]<br />
* [[Avogadro Procedure| Avogadro]]<br />
* [[BuildQsar Procedure| BuildQSAR]]<br />
* [[ChemDraw Procedure| ChemDraw]]<br />
* [[ChemSketchProcedure| ChemSketch]]<br />
* [[Dragon5 Procedure| Dragon5]]<br />
* [[Generating Descriptors Workflow]]<br />
* [[HyperChem]]<br />
* [[Jupyterlab anaconda| JupyterLab]]<br />
* [[OpenBabelGUI]]<br />
* [[QSARINS Procedure| QSARINS]]<br />
* [[VASP and VMD on CCAST| VASP]]<br />
* [[WEKA Steps for Loading Data| WEKA]]<br />
</div><br />
</div><br />
</div><br />
<br />
__NOEDITSECTION__<br />
__NOTOC__</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Main_Page&diff=126Main Page2022-10-19T20:28:27Z<p>Sysadmin: Updated front page</p>
<hr />
<div><div><br />
=Welcome to the Rasulev Research Group Wiki!=<br />
==This page hosts the tutorials and procedures for the Computational Polymer Science and Cheminformatics Group at North Dakota State University==<br />
This page implements MediaWiki, and provides a platform to organize knowledge and make it available to lab collaborators and students. <br />
<div style="clear: both;"></div><br />
<div id="audiences" class="mainpage_row"><br />
<div class="mainpage_box"><br />
<h3><span class="header_icon" aria-hidden="true" role="presentation">[[File:OOjs UI icon chem.svg|20px|middle|link=]]</span><span>Lab Procedures and Software Tutorials</span></h3><br />
<div> <br />
* [[ADF ReaxFF]]<br />
* [[Avogadro Procedure| Avogadro]]<br />
* [[BuildQsar Procedure| BuildQSAR]]<br />
* [[ChemDraw Procedure| ChemDraw]]<br />
* [[ChemSketchProcedure| ChemSketch]]<br />
* [[Dragon5 Procedure| Dragon5]]<br />
* [[Generating Descriptors Workflow]]<br />
* [[HyperChem]]<br />
* [[Jupyterlab anaconda| JupyterLab]]<br />
* [[OpenBabelGUI]]<br />
* [[QSARINS Procedure| QSARINS]]<br />
* [[VASP and VMD on CCAST| VASP]]<br />
* [[WEKA Steps for Loading Data| WEKA]]<br />
</div><br />
</div><br />
</div><br />
<br />
__NOEDITSECTION__<br />
__NOTOC__</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=VASP_and_VMD_on_CCAST&diff=125VASP and VMD on CCAST2022-09-26T21:32:19Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
{|<br />
|<br />
'''Login''' '''Photon''':<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
ssh [mailto:kilin@photon.chem.ndsu.nodak.edu <u>kilin@photon.chem.ndsu.nodak.edu</u>]<br />
<br />
cd /scratch001/Chem_476/CHEM676_2019/Meade/DIMER<br />
| <br /><br />
|-<br />
|<br />
'''Login''' '''Cori''':<br />
<br />
<br /><br />
| Have to be in Photon first then login to Cori<br />
|-<br />
|<br />
ssh -i ~/.ssh/nersc cori.nersc.gov<br />
| <br /><br />
|-<br />
|<br />
cd /global/cfs/cdirs/m1251/vasp/CHEM676/meade<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
'''Misc. Tools'''<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| cd foldername<br />
| lets you go into a folder<br />
|-<br />
| cd ..<br />
| Backs you out of folder you are in<br />
|-<br />
| cd -<br />
| Goes back to previous directory<br />
|-<br />
| cd ~<br />
| Goes to home directory<br />
|-<br />
| freenodes<br />
| in ccast<br />
|-<br />
| vi filename<br />
| Lets you go into file to edit<br />
|-<br />
| i<br />
| When in “vi” this initiates the editing<br />
|-<br />
| :wq<br />
| This lets you stop editing<br />
|-<br />
| “Ctrl+c” press on keyboard, not a typing thing<br />
| Lets you stop any current running process<br />
|-<br />
| pwd<br />
| Gives you current directory location<br />
|-<br />
| wc<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| ls<br />
| Lists what is in the current directory<br />
|-<br />
| ls -lt<br />
| Lists what is in the current directory sorting from newest to oldest<br />
|-<br />
| ls –lt W*<br />
| Lists everything that starts with a W sorting from newest to oldest. (Can be used for other letters as well)<br />
|-<br />
| ls -lisanh<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisanh filename<br />
| gives information on the specific file<br />
|-<br />
| ls -lisan<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisan filename<br />
| gives information on the specific file<br />
|-<br />
| ls -a<br />
| shows all files, including hidden ones<br />
|-<br />
| babel –ixyz file.xyz –opdb file.pdb xyz to pdb<br />
|<br />
Using babel, converts the input file of .xyz to outputfile of .pdb<br />
<br />
That is where the –ixyz and –opdb comes from<br />
|-<br />
| “shift+alt+left click”<br />
| For slecting just comlumn or just row information you want exactly<br />
|-<br />
| egrep '163|166' OS_STRENGTH<br />
| grep technique<br />
|-<br />
| more filename<br />
| Lets you see what is in the file<br />
|-<br />
| more ~/bin/filename<br />
| Lets you see the file in a location not in your current directory.<br />
|-<br />
| more filename | grep ### or value | grep again if desired<br />
| shows file and grabs lines with same input you put after the grep command<br />
|-<br />
| cp /Location/filename .<br />
| Copies a file from a different directory location and input it into the current directory. Don’t forget the PERIOD!<br />
|-<br />
| P /Location/filename ./newfilename<br />
| Copies a file from a different directory location and input it into the current directory and rename it in one step<br />
|-<br />
| scp sourceUser:/sourceLocation/filename .<br />
| Takes file from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/* .<br />
| Takes ALL files from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/*.filetype .<br />
| Takes ALL files with that specific file type from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp * destinationUser:/destinationLocation<br />
| Sends all files from source directory to destination directory. Must be in source directory when typing this line<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/DIMabsorptionspec<br />
| Example of scp<br />
|-<br />
| sbatch corifilename<br />
| Lets you submit a file in cori using a corifile made by others<br />
|-<br />
| grep specificdetail filename<br />
| Grabs the specific detail from the file you chose<br />
|-<br />
| sort -n -r filename<br />
| sorts the file in ascending order<br />
|-<br />
| babel BondLength.xyz -O out.xyz --separate<br />
| THIS IS CORRECT WAY THE SEP WORD IS SPELLED INCORRECTLY BUT IS CORRECT FOR BABEL<br />
|-<br />
| awk '{ if($4 &gt;= 4.0000 &amp;&amp; $4&lt;=5.0000) print }' OS_STRENGTH1<br />
| lets you get lines of data that contain the range from 4-5 from column 4<br />
|-<br />
| users<br />
| lets you see what users are on<br />
|-<br />
| clear<br />
| lets you clear screen<br />
|-<br />
| who<br />
| who is on and more detail about them<br />
|-<br />
| w<br />
| all the people and differing details about them<br />
|-<br />
| id<br />
| lets me see what groups I am in<br />
|-<br />
| freenodes<br />
| lets you see if nodes are available or not<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6335cce9234bc4ef.png|439x225px]]<br />
|<br />
|-<br />
| pbsnodes –avSj | sort -n<br />
| shows available nodes, the | sorts it into order<br />
|-<br />
| pbsnodes –avSj | sort –n | more<br />
| does this in page format rather than all at once<br />
|-<br />
| module avail<br />
| what software is available<br />
|-<br />
| $HOME<br />
| lets you go to your login directory “home” directory<br />
|-<br />
|<br />
thunder.ccast.ndsu.edu<br />
<br />
meade.erickson<br />
| to get into putty<br />
|-<br />
| C:\Users\meade.erickson\Documents\OriginLab\User Files\<br />
| Origin themes location<br />
|-<br />
| '''Explanations'''<br />
| <br /><br />
|-<br />
| fermi<br />
| fermi energy is referring to the energy difference between the highest and lowest occupied single-particle states. ([https://en.wikipedia.org/wiki/Fermi_energy <u>https://en.wikipedia.org/wiki/Fermi_energy</u>])<br />
|-<br />
| '''Explanations Input Files:'''<br />
| <br /><br />
|-<br />
| INCAR<br />
| It is the central input file of VASP. It determines &quot;what to do and how to do it&quot;. ([https://www.vasp.at/wiki/index.php/INCAR <u>https://www.vasp.at/wiki/index.php/INCAR</u>])<br />
|-<br />
| POSCAR<br />
| <br /><br />
|-<br />
| POTCAR<br />
| <br /><br />
|-<br />
| KPOINTS<br />
| <br /><br />
|-<br />
| '''Explanations Output Files:'''<br />
| <br /><br />
|-<br />
| CONTCAR<br />
|<br />
Contains actual coordinates, velocities, and predictor corrector coordinates. 1<sup>st</sup> block is lattice parameters and atom coordinates. 2<sup>nd</sup> block Initial Velocites for atoms. 3<sup>rd</sup> block Predictor-corrector coordinates.<br />
<br />
Written at end of each job. Used as the POSCAR file for next MD job.<br />
|-<br />
| WAVECAR<br />
| Provides wavefunctions. we usually use it for spectrum data ([https://www.vasp.at/wiki/index.php/WAVECAR <u>https://www.vasp.at/wiki/index.php/WAVECAR</u>])<br />
|-<br />
| OS_STRENGTH<br />
| Information about the OS_STRENGTH<br />
|-<br />
|<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8bf47c7304419724.png|354x154px]]<br />
|<br />
From Wikipedia<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
In spectroscopy, '''oscillator strength''' is a dimensionless quantity that expresses the probability of [https://en.wikipedia.org/wiki/Absorption_(electromagnetic_radiation) <u>absorption</u>] or [https://en.wikipedia.org/wiki/Emission_spectrum <u>emission</u>] of [https://en.wikipedia.org/wiki/Electromagnetic_radiation <u>electromagnetic radiation</u>] in transitions between [https://en.wikipedia.org/wiki/Energy_level <u>energy levels</u>] of an atom or molecule<sup>[</sup>[https://en.wikipedia.org/wiki/Wikipedia:Accuracy_dispute#Disputed_statement <sup>''<u>dubious</u>''</sup>]<sup>'' – ''</sup>[https://en.wikipedia.org/wiki/Talk:Oscillator_strength#Wrong_definition_oscillator_strength <sup>''<u>discuss</u>''</sup>]<sup>]</sup>.<sup><u>[https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Demtr%C3%B6der2003-1 [1]][https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Robinson1996-2 [2]]</u></sup> The oscillator strength can be thought of as the ratio between the quantum mechanical transition rate and the classical absorption/emission rate of a single electron oscillator with the same frequency as the transition.<br />
|-<br />
| POTIM (in INCAR file)<br />
| timestep in femtoseconds. So POTIM=1 is 1 femtosecod<br />
|-<br />
| '''Density of States'''<br />
| <br /><br />
|-<br />
| To get DOS you need to run the correct calculations in VASP using 4 files. INCAR, POSCAR, POTCAR, OUTCAR<br />
| <br /><br />
|-<br />
| Once you get output from your submitted information to cori or photon you can follow these steps to get Density of states<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| Grabs the number of bands (we need this for later) (STATES)<br />
|-<br />
| vi input_overlap<br />
| editing input_overlap<br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| Number of Bands from the grep NBA OUTCAR line<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| quitting editing input_overlap<br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| getting energy pop from different location<br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| wc STATES<br />
| <br /><br />
|-<br />
| grep NELE OUTCAR<br />
| record number of electrons<br />
|-<br />
| grep NIO OUTCAR<br />
| record number of ions<br />
|-<br />
| grep E-fer OUTCAR<br />
| record the three values and their names. If this line doesn’t work the try next line)<br />
|-<br />
| grep ferm OUTCAR | tail –n 1<br />
| record the three values and their names<br />
|-<br />
| ~/bin/DRESS_DOS_NORM<br />
| Executes the DOS code from different file location<br />
|-<br />
|<br />
Number of states from grep NBA OUTCAR<br />
| grep NBA OUTCAR line<br />
|-<br />
| -20<br />
| <br /><br />
|-<br />
| 20<br />
| <br /><br />
|-<br />
|<br />
E-fermi info from grep E_fermi<br />
<br />
<br /><br />
|<br />
To literally line up DOS and STATES, you may need to type in 0 for both fermi and alpha+beta.<br />
<br />
<br />A more standard way is to use the middle point energy between HOMO and LUMO i.e. (E_HOMO+E_LUMO)/2 for fermi and 0 for alpha+beta. In this way, the mid point of DOS is at zero, anything below zero is occupied and above zero is unoccupied.<br />
|-<br />
|<br />
alpha+beta info from grep E_fermi<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| .01<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
|<br />
Number of ions from grep NIO OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
Number of electrons from grep NELE OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| 3<br />
| <br /><br />
|-<br />
|<br />
gnuplot ~/bin/gnuprog_DOS.fermi<br />
<br />
<br /><br />
| Plotting the DOS (This is done in NERSC or PHOTON)<br />
|-<br />
|<br />
ps2pdfwr DOS.ps<br />
<br />
<br /><br />
| Converting DOS file from .ps to .pdf<br />
|-<br />
|<br />set out MEADESDOS.ps MAY NOT NEED THIS NOT SURE YET<br />
set style line 4 lw 4 lc -1; unset key; plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
set style line 4 lw 4 lc -1<br />
<br />
unset key<br />
<br />
plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4;<br />
<br />
set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0<br />
<br />
set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0<br />
<br />
set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0<br />
<br />
set tics font &quot;,30&quot;<br />
<br />
set ytics offset 0,0,0<br />
<br />
set xtics offset -0.4,-1,0<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.518567):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.518567):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.173350):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.17335):4 w l title &quot;Wet Monomer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Wet Monomer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
|<br />
I am initially specifying a style that I will be using. In this case chose style 4 and am editing it to what I want.<br />
<br />
lw stands for linwidth, lc line color, not sure where the library is but lc -1 is black, 0=red I think and find out other through trial and error<br />
<br />
lw is just multiplyer, so lw 4 is 4 times larger than default.<br />
<br />
unset key gets rid of key,<br />
<br />
[#,#] is the range of my x axis graph.<br />
<br />
($#-###): $# is the column I want to take data from which is being subtracted by the amount of shift I am giving to the data (we want data to be centered at 0 sometimes and so for DOS we take the middle between the start of non occupied plus the end of occupied and divide by 2 to get our shift. (Notoccupied + occupied)/2 ) and the number after the “:” is the column I am correlating the X and Y axis with. So when we plot stuff we need to specify what is the x and what is the y axis.<br />
<br />
not sure exactly what the “w i” portion does but it helps us change from points to lines and then is what fills in the red portion of our graph. (The “i” stands for impulses. Add an extra “lw 2’ to increase the line width as shown in green text.) “w l” does the same thing but DOESN”T fill out the the electron density portion.<br />
<br />
xlabel and y labels are the axis titles and offsets help position them due to the changing sizes of the axes and tics.<br />
<br />
Tics are the numbers on the Y and x axis so ytics offsets them and xtics does the horizontal axis change.<br />
<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
| '''Absorption Spectrum - Transition Energy eV – Wavelength nm'''<br />
| <br /><br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
|<br />
copying INCAR-gm into the directory as long as you are in it<br />
<br />
MAKE SURE<br />
<br />
LWAVE= .TRUE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori07:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec/TEST<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Or make sure the INCAR has this Ag+Carboxyl<br />
<br />
#ISIF=7<br />
<br />
#PREC = HIGH<br />
<br />
ENCUT= 300<br />
<br />
IBRION =3<br />
<br />
POTIM=0.2<br />
<br />
NSW = 30000<br />
<br />
ALGO = NORMAL #48<br />
<br />
ISMEAR = 0<br />
<br />
SIGMA = 0.01<br />
<br />
LREAL = A<br />
<br />
#ISTART = 0<br />
<br />
LWAVE = .TRUE.<br />
<br />
LCHARG = .FALSE.<br />
<br />
EDIFF=1E-6<br />
<br />
EDIFFG=-0.05<br />
<br />
LVTOT = .FALSE.<br />
<br />
#LVDW=.TRUE.<br />
<br />
#SPRING = -5<br />
<br />
#LCLIMB = .TRUE.<br />
<br />
#IMAGES = 7<br />
<br />
#IOPT=7<br />
<br />
#ICHAIN=0<br />
<br />
NELM=40<br />
<br />
#ISPIN=2<br />
<br />
ISYM=0<br />
<br />
NPAR=4<br />
<br />
<br /><br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|345x215px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| It is necessary to have the ‘spectrum’ file made which comes from a healthy WAVECAR which is used to generate OS_STRENGTH. After OS_STRENGHT we make the spectrum ~/bin/SPECTRUM_imp3 file.<br />
| <br /><br />
|-<br />
| Assuming you have conducted a calculation which generated a healthy WAVECAR file.<br />
| <br /><br />
|-<br />
| If healthy WAVECAR exists then…<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| record Number of bands<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
|<br />
1<br />
<br />
Number of Bands<br />
<br />
1<br />
| 36740210<br />
|-<br />
|<br />
‘esc’ key<br />
<br />
:wq<br />
| <br /><br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| <br /><br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| ./osc_str.exe<br />
| This makes the OS_STRENGTH (oscillator strength)<br />
|-<br />
| grep '2.0000 0.0000' OS_STRENGTH &gt; OS_STRENGTHvc<br />
|<br />
Takes the orbital pairs from 2.0000 to 0.0000 into new file.<br />
<br />
<br /><br />
|-<br />
| cp OS_STRENGTH OS_STRENGTH1<br />
| making sure I have original OS_STRENGTH<br />
|-<br />
| cp OS_STRENGTHvc OS_STRENGTH<br />
| making sure I have file OS_STRENGvc kept for 2.0000 – 0.0000 but it will be our OS_STRENGTH<br />
|-<br />
| wc OS_STRENGTH<br />
| Record First number<br />
|-<br />
| ~/bin/SPECTRUM_imp3<br />
| <br /><br />
|-<br />
|<br />
Number from OS_STERNEGTH<br />
<br />
.1<br />
<br />
10<br />
<br />
.001<br />
<br />
HOMO<br />
|<br />
Should be able to find HOMO from STATES file (or energy_pop)<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Can change the “.001” to “0.1 or 0.01” to smooth out the line<br />
|-<br />
| a ‘spectrum’ file should have been made. This holds all the spectrum data. The spectrum can be visualized using these commands in Cori. If you want to visualize in photon then use the gnuplot information below<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE<br />
|<br />
IN CORI YOU CAN USE THESE<br />
<br />
THEN SCP them to photon, then the desktop!<br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE_nm_200_800<br />
|<br />
|-<br />
| ps2pdfwr SPE.ps<br />
|<br />
|-<br />
| ps2pdfwr SPE_nm.ps<br />
|<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec<br />
| example make sure you are in the directory of the files you want to send. Send it to a premade directory in photon<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
|<br />
FOR PHOTON USE THESE STEPSWe are plotting the information of spectrum to get the spectrum in wavelength of nanometers nm and transition energy in electron volts eV<br />
| <br /><br />
|-<br />
| more spectrum<br />
|<br />
(Be sure to be in correct directory)<br />
<br />
Column 1 is the spectrum in wavelength of nm<br />
<br />
column 2 is the spectrum in transition energy of eV<br />
<br />
column 3 is absorbance/absorption in a.u. (absoption units). Absorbance goes with wavelength, absorption goes with transition energy<br />
<br />
column 4 idk what this is.<br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| plot ‘spectrum’ 2:3<br />
| This gives you transition energy in x axis and absorption in a.u for yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [0:10] 'spectrum' using 2:3 w l title &quot;Transition Energy&quot; ls 4; set xlabel 'Transition Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorption a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| This is transition energy in x axis and absorption in yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [200:800] 'spectrum' using 1:3 w l title &quot;Absorption Spectrum&quot; ls 4; set xlabel 'Wavelength, nm' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorbance, a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| nm in x axis and absorbance in yaxis<br />
|-<br />
| '''UV-Vis Absorption Spectrum'''<br />
| <br /><br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
We did one calculation already but we need to do a “geometry optimization” for the next one.<br />
<br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
| copying INCAR-gm into the directory as long as you are in it<br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|352x219px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| '''Generating CANDIDATES file for Photofragmentation'''<br />
| <br /><br />
|-<br />
| This is after calculations have been ran for MD???<br />
|<br />
Watch this first‼ Part 2 of Dr.Kilins video for how to generate the CANDIDATES file.<br />
<br />
[https://www.youtube.com/watch?v=NIrs1yXJj1Y&feature=youtu.be <u>https://www.youtube.com/watch?v=NIrs1yXJj1Y&amp;feature=youtu.be</u>]<br />
|-<br />
| We want to get information from the WAVECAR that can help us determine the “electron hopping” so we have to find the OS_STRENGTH (oscillator strength) that is high. Since higher OS_Strength the higher the probability that given pair of orbitals is excited. STATES file should have been generated before I think.<br />
|<br />
Watch this second‼ Part 1 of Dr. Kilins video for what to do after CANDIDATES file was generated and explains what is in it.<br />
<br />
[https://www.youtube.com/watch?v=A8BXT-FnuxM&t=212s <u>https://www.youtube.com/watch?v=A8BXT-FnuxM&amp;t=212s</u>]<br />
|-<br />
| grep –A 700 fermi OUTCAR | tail –n 700 &gt; STATES1<br />
| Fermi information from OUTCAR into a file called STATES 1. Not sure what the –A or 700 mean. But the tail is choosing a specific section of the data, the “tail” end of the chosen information from the OUTCAR file.<br />
|-<br />
| more STATES1<br />
| Determine HOMO and LUMO, HOMO is the highest orbital with a 2, LUMO is the first 0. In my case HOMO is 112 and LUMO is 113<br />
|-<br />
| NOT SURE HOW HE GENERATED STATES FILE<br />
| <br /><br />
|-<br />
| head STATES<br />
| Record the lowest number, in my case it is 81 for DIMER<br />
|-<br />
| tail STATES<br />
| Record the Highest number, in my case it is 160 for DIMER<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| 160<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| more ~/bin/ADVICE.spectrum<br />
| This will make spectrum. This will show 3 lines of code. Literally copy one line and press enter starting with the “module swap PrgEnv-intel PrgEnv-gnu”<br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| copying templates and inputting them into the directory you are in<br />
|-<br />
| ./osc_str_CHEM676.exe<br />
| This reads binary WAVECAR file, Reads input_overlap, creates Oscillator strength<br />
|-<br />
| ls –lt | head<br />
| This should show a fresh OS_STRENGTH was made<br />
|-<br />
| grep ‘2.0000 0.0000’ OS_STRENGTH &gt; OS_NEW<br />
| In the OS_STRENGTH file we will see lines of no transition from occupied to unnoccupied. What we want to see is the lines that have an occupied originally (2.0000) and then are unoccupied (0.0000). This shows excitation. The lines that meet this requirement are then inputted into OS_NEW<br />
|-<br />
| head OS_NEW<br />
| This should show a range of orbitals that have been occupied and are now unoccupied. With the differing orbitals involved. Columns 1 and 2 are the initial and final orbitals that are involved with the “hopping” of electrons. The 3rd column is oscillator Strength. 4th Column is Transition energy.<br />
|-<br />
| mv OS_NEW OS_STRENGTH<br />
| We don’t need all OS_STRENGTH data, just the stuff we “grepped” in OS_NEW. This overrides OS_STRENGTH with information from OS_NEW.<br />
|-<br />
| wc OS_STRENGTH<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| head –n 30 OS_STRENGTH<br />
| In this particular case we only want the top 30 rather than the original value the first column gave was. In my case it was about 1536.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4<br />
| To sort in ascending order of transition energy. 4 is the 4<sup>th</sup> column. I don’t know what –gk means.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4 &gt; CANDIDATES<br />
| Inputs this information into CANDIDATES file<br />
|-<br />
| '''Orbital Visualization'''<br />
| <br /><br />
|-<br />
| I don’t have all information available here, for example I personally didn’t make the PARCHG files.<br />
| <br /><br />
|-<br />
|<br />
Kilins Guidance to make PARCHG files… I think<br />
<br />
Dear Meade,<br />
<br />
below are steps I did for creating orbitals files on nersc.<br />
<br />
I will let you know when they are ready and instruct on how to visualize.<br />
<br />
slides are good, under given limits.<br />
<br />
In the given time, you may add conclusions as you see them..<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; mkdir ORBITALS<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp WAVECAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp CONTCAR ORBITALS/POSCAR<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp POTCAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp *.sh ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp KPOINTS ORBITALS/<br />
<br />
&gt; cd ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; grep NBA ../OUTCAR<br />
<br />
k-points NKPTS = 1 k-points in BZ NKDIM = 1 number of bands NBANDS= 160<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; cp ~/bin/INCAR/INCAR-pc ./INCAR<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; tail INCAR<br />
<br />
IBRION=2 #conjugate-gradient algorithm used to relax ions (bad<br />
<br />
NSW=0 #number of ionic steps<br />
<br />
POTIM= .2 #time step in fs<br />
<br />
EDIFFG=-0.001 #minimum energy difference between ionic iterations<br />
<br />
EINT= -4.4 -0.5<br />
<br />
EINT= -6 2 <br />
<br />
LVTOT = .TRUE.<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; sbatch cori_debug.sh <br />
<br />
Submitted batch job 32413471<br />
<br />
many thanks for the new version!<br />
<br />
The main question in your email is orbitals with VASP:<br />
<br />
a. generate or use premade healthy WAVECAR <br />
<br />
(it is an analog of .chk in gaussian)<br />
<br />
b. in a new directory, use template of the INCAR for orbitals and run vasp once again.<br />
<br />
the templates are named ~/bin/INCAR/INCAR_pc and similar names.<br />
<br />
(it is an analog of the cubegen procedure in gaussian)<br />
<br />
c. upon successful run, there appear files nammed PARCHG.XXXX.ALLK, where XXXX stands for the index of orbital, e.g. PARCHG..01112.ALLK<br />
<br />
(those files are analogs of the .cube files in gaussian. note that vasp DOES not generate literal .cube files!)<br />
<br />
d. download these files and open then in the VMD - the same software you have used for the trajectories.<br />
<br />
I will have a look on the slides and let you nko if any quicl changes are needed or we can go ahead with this version<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
<br /><br />
|<br />
|-<br />
| Step 2. Changing PARCHG Files to cube files (For Orbital Visualization)<br />
| From what I have seen we need cube files to show in guassview AND VMD<br />
|-<br />
| Once obtained PARCH.####.ALLK Files then use following command in PHOTON to convert the orbitals you care about to cube!<br />
| <br /><br />
|-<br />
| ~/vtstools/chg2cube.pl filename<br />
| <br /><br />
|-<br />
| 3<br />
| selecting PARCHG<br />
|-<br />
| 1 6 7 8<br />
| (These are the atomic numbers for the atoms in your system. In my case I have Hydrogen, Nitrogen, Oxygen, and Carbon<br />
|-<br />
| Your cube file is then generated<br />
| <br /><br />
|-<br />
| SEND cube files, POSCAR (OR CONTCAR), CHGCAR FILES TO DESKTOP<br />
| <br /><br />
|-<br />
| Step 3. Using VMD to generate orbitals<br />
| <br /><br />
|-<br />
| Open VMD<br />
| <br /><br />
|-<br />
|<br />
In the VMD Main window go to “File” “new Molecule” Upload your<br />
<br />
1.POSCAR (OR POSSIBLY CONTCAR): By selecting VASP_POSCAR under the “Determine file type:”<br />
<br />
2. CHGCAR: By selecting VASP_CHGCAR under the “Determine file type:”<br />
<br />
3. USE THIS ONE PARCHG: By selecting VASP_PARCHG under the “Determine file type:” (Even though the file ends in .cube we are classifying it as PARCHG)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b0283f3c67e04e7d.png|219x134px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_eae9e77b87b8627b.png|222x103px]]<br />
|-<br />
|<br />
After all three are uploaded,<br />
<br />
in the VMD Main window go to “Graphics” and then “Representations”<br />
| <br /><br />
|-<br />
|<br />
In the Graphical Representations window follow these three examples.<br />
<br />
<br />(Change the file you are using by the dropdown menu of “selected Molecule”)<br />
|<br />
The red lettering means that PARCHG Isosurface isn’t being shown. You can double click on the red lettering to make it black and showing on the OpenGL Display Window. Double click again to revert to black.<br />
<br />
Parchrg can be shown and can be visualized in conjungtion with the CHGCAR file<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_e7a558f4dc169c4e.png|121x260px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b15ffb6b911b765.png|124x266px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3734231f026ab64e.png|124x265px]]<br />
|<br />
|-<br />
|<br />
-You can change the iso value in CHGCAR to get varying orbital visualizations.<br />
<br />
-Play with it.<br />
<br />
-Change the color by changing the ColorID to different ones.<br />
<br />
-(Range in CHGCAR doesn’t matter)<br />
|<br />
For Monomer Project Show<br />
<br />
all index 52 47 29 44 38 28 18 56 64<br />
<br />
H52<br />
<br />
H47<br />
<br />
H29<br />
<br />
H44<br />
<br />
H38<br />
<br />
H28<br />
<br />
H18<br />
<br />
H56<br />
<br />
H64<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c5f67d67a9a89fa7.png|355x180px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d4c6c68549d8e025.png|357x209px]]<br />
| <br /><br />
|-<br />
| To get rid of opaque water molecuels make both of these in “Graphics” “Materials” 123 is for faded bods and 124 is for the orbitals<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8d905765b5183c3.png|142x229px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_449e5be8bb4a3541.png|139x222px]]<br />
|<br />
|-<br />
| For my Monomer_H2O first frame hide these atoms if you don’t want in view<br />
| <br /><br />
|-<br />
|<br />
O108<br />
<br />
O99<br />
<br />
O94<br />
<br />
O101<br />
<br />
O81<br />
<br />
O91<br />
<br />
O90<br />
<br />
O86<br />
<br />
O82<br />
<br />
O87<br />
<br />
O79<br />
<br />
<br />H73<br />
<br />
H45<br />
<br />
H62<br />
<br />
H51<br />
<br />
H58<br />
<br />
H50<br />
<br />
H48<br />
<br />
H14<br />
<br />
H22<br />
<br />
H36<br />
<br />
H42<br />
<br />
H31<br />
<br />
H20<br />
<br />
H23<br />
<br />
H35<br />
<br />
H30<br />
<br />
H54<br />
<br />
H59<br />
<br />
H25<br />
|<br />
all not index ### ### ### ###...<br />
<br />
all not index 108 99 94 101 81 91 90 86 82 87 79 73 45 62 51 58 50 48 14 22 36 42 31 20 23 35 30 54 59 25<br />
<br />
<br />backbone <br />
<br />
index 97 103 11 9 10 13 12 96 1 84 78 3 4 100 77 0 8 2 5 6 7<br />
|-<br />
| '''These are for the figure first frame for still seeing the water atoms but hazy'''<br />
| <br /><br />
|-<br />
|<br />
H22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46<br />
<br />
<br />O90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
| 22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46 90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
|-<br />
| '''Index for Diagram for pics Middle Frame for H20 molecules'''<br />
| <br /><br />
|-<br />
|<br />
H63<br />
<br />
H33<br />
<br />
H17<br />
<br />
H21<br />
<br />
60<br />
<br />
68<br />
<br />
42<br />
<br />
38<br />
<br />
34<br />
<br />
50<br />
<br />
72<br />
<br />
73<br />
<br />
16<br />
<br />
61<br />
<br />
55<br />
<br />
71<br />
<br />
59<br />
<br />
69<br />
<br />
40<br />
<br />
14<br />
<br />
39<br />
<br />
74<br />
<br />
67<br />
<br />
53<br />
<br />
49<br />
<br />
41<br />
<br />
52<br />
<br />
27<br />
<br />
15<br />
<br />
31<br />
<br />
23<br />
<br />
54<br />
<br />
62<br />
<br />
32<br />
<br />
64<br />
<br />
30<br />
<br />
26<br />
<br />
57<br />
<br />
37<br />
<br />
51<br />
<br />
58<br />
<br />
70<br />
<br />
43<br />
<br />
25<br />
<br />
35<br />
<br />
46<br />
<br />
45<br />
|<br />
O80<br />
<br />
87<br />
<br />
99<br />
<br />
81<br />
<br />
88<br />
<br />
105<br />
<br />
104<br />
<br />
91<br />
<br />
90<br />
<br />
106<br />
<br />
97<br />
<br />
108<br />
<br />
102<br />
<br />
89<br />
<br />
93<br />
<br />
101<br />
<br />
92<br />
<br />
107<br />
<br />
78<br />
<br />
109<br />
<br />
86<br />
<br />
83<br />
<br />
94<br />
<br />
63 33 17 21 60 68 42 38 34 50 72 73 16 61 55 71 59 69 40 14 39 74 67 53 49 41 52 27 15 31 23 54 62 32 64 30 26 57 37 51 58 70 43 25 35 46 45 80 87 99 81 88 105 104 91 90 106 97 108 102 89 93 101 92 107 78 109 86 83 94<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_783e625ce0b8a321.png|182x247px]]<br />
|-<br />
| '''Preparation of TDESMD Orbital Files and Job Submission'''<br />
| <br /><br />
|-<br />
| mkdir TDESMD_111_114<br />
| We will now use the information of the CANDIDATES file and create new folder for each scenario. You create a file titled using the orbitals involved. In my case the first scenario is with orbitals 111 and 114<br />
|-<br />
| cd TDESMD_111_114<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/INCAR_EXCITED<br />
| Taking INCAR_EXCITED file from location. In my case it is here.<br />
|-<br />
| vi INCAR_EXCITED<br />
| Edit the INCAR_EXCITED file to make it look like<br />
|-<br />
| i<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_79a32df6f9f33bb0.png|303x201px]]<br /><br />
|-<br />
| <br /><br />
|<br />
Change FERWE=#### to match the total number of orbitals/bands. In this example we have a total number of 160 bands/orbitals.<br />
<br />
<br />
The first segment of FERWE, “110*1” says we have the first 110 orbitals occupied. The second portion is orbital number 111 which we have listed as a “0” signifying this orbital is unnoccopied for our job submission. Then orbital number 112 is occupied with “1”. Then orbital number 113 is unoccupied with “0”. Then orbital number 114 is occupied with “1”. The last part of “46*0” is talking about the other orbitals not mentioned. These orbital should add up to 160. These numbers change with the differing orbitals designating in the CANDIDATES FILE.<br />
<br />
The main thing that is changing is the position of one occupied orbital and one unoccupied orbital when the compound is in ground state. In this example number 111 is usually occupied in ground state and we are moving the electron to 114 which is unoccupied in ground state.<br />
|-<br />
| <br /><br />
| ONE more example is as follows. With TDESMD_102_115<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_51e84877e4374ca6.png|300x199px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POSCAR ./CONTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/KPOINTS<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/debug1.sh<br />
| <br /><br />
|-<br />
| sbatch debug1.sh<br />
| <br /><br />
|-<br />
| '''Creating Videos from movieAll.xyz files and uploading them on youtube'''<br />
| <br /><br />
|-<br />
| scp –r TDESMD_###_### DestinationUser:/DestinationLocation<br />
|<br />
Starting in cori<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Using this to send files from cori to photon. We specifically want the movieALL.xyz files.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
This is sending the WHOLE directory and everything it contains<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| mkdir TDESMD_###_###<br />
|<br />
In DESKTOP<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
generate TDESMD_###_### directories<br />
|-<br />
| scp kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_##_###/movieALL.xyz .<br />
| WHILE IN THE TDESMD DIRECTORY THAT YOU WANT TO BE THE DESTINATION take “movieALL.xyz” from photon and put into specific directory<br />
|-<br />
| <br /><br />
| Once the file is in your desktop you can go to your windows folders and open it to check if it is there.<br />
|-<br />
| In windows search bar<br />
| <br /><br />
|-<br />
| VMD<br />
|<br />
type vmd in windows search bar and open it.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
VMD will open with multiple screens.<br />
<br />
<br /><br />
|-<br />
| We will focus on this screen first to open all the extra screens we need. (VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_5f31d31c2c72c623.png|244x114px]]<br />
|-<br />
| Press “File”, “New Molecule,”<br />
| <br /><br />
|-<br />
| On the window that comes up (Molecule File Browser) Press “Browse” and find the file you would like. Find the file you want. In this example it is an xyz file.<br />
| <br /><br />
|-<br />
| Once A file is picked then press “Load” in the Molecule File Browser window.<br />
| <br /><br />
|-<br />
| Going back to VMD Main<br />
| <br /><br />
|-<br />
|<br />
Press “Graphics” and Representations to open a long slender screen.(VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c8cfeb66ca7723b0.png|225x487px]]<br />
|-<br />
| Press the “Create Rep” Button (Graphical Representation window)<br />
| <br /><br />
|-<br />
| With one of the “Reps” highlighted in green, in the drawing Method switch from Lines to VDW<br />
| <br /><br />
|-<br />
|<br />
Change the Sphere scale to 0.2.<br />
<br />
Resolution can stay at default or can be raised if desired.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6ceb4d7e0c7c142a.png|157x340px]]<br />
|-<br />
|<br />
Click on the other line that still says “LINES” and in the Drawing Method change it to Dynamic Bonds<br />
<br />
Change Distance Cutoff to 1.6<br />
<br />
Bond Radius to 0.1<br />
<br />
And Resolution can stay at default or be increased<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_dba6c2b704e3401a.png|158x337px]]<br />
|-<br />
| Go back to VMD Main<br />
| <br /><br />
|-<br />
| Go to “Extensions,” “Visualizations,” and “Movie maker”<br />
| <br /><br />
|-<br />
| Go to the VMD Movie Generator window<br />
| <br /><br />
|-<br />
| In the box of “Name of movie” type the TDESMD_##_### you have.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|214x167px]]<br />
|-<br />
| “Set Working Directory“ choose the destination directory you want the movie to go to.<br />
| <br /><br />
|-<br />
| “Rotation Angle” I put it to 0 but you can put it to whatever.<br />
| <br /><br />
|-<br />
| “trajectory step size” 1<br />
| <br /><br />
|-<br />
| “Movie Duration” This depends on your frames from the (VMDMain) Usually 1000 frames is about 20-30 seconds if the speed (step) is set to 1. But By changing the speed of the video we should make the total video length to 30-45 seconds long.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|232x181px]]<br />
|-<br />
| Back to VMD Main window<br />
| <br /><br />
|-<br />
| Press “Mouse,” “Label,” “Bonds”<br />
| <br /><br />
|-<br />
| click on two neighboring atoms to get the atom name and bond length<br />
| <br /><br />
|-<br />
|<br />
IN VMD Main, go to “Graphics” and “Labels.” select “atoms” in the drop down menu on the “Labels” window.<br />
<br />
Select all atoms by clicking on the top atom and then “SHIFT+left click” the bottom atom to select all atoms<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
While all atoms are selected (highlighted in green). Go to “Properties” and in the “Format” box type %e%i.<br />
<br />
(The %e gives the element and the %i gives the index number)<br />
<br />
Some information can be presented by simply having your cursor over the %e%i and a yellow box will pop up for some info.<br />
<br />
You can move the lettering of each atom by moving the offset information.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_46eab8f828aa40bc.png|215x201px]]<br />
|-<br />
| Click the global properties and change text size and text thickness. ( all of this can be see in the (VMD1.9.3 OpenGL Display) [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_260ebe13a2eda05.png|162x114px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3968014bc41ed6a4.png|224x212px]]<br />
|-<br />
| in VMD movie Generator press “make movie” and quickly in VMD main press the play button.<br />
| <br /><br />
|-<br />
|<br />
Once the video is done a popup should be produced asking where to find videomach. Here is the directory for my computer.<br />
<br />
C:\Program Files (x86)\VideoMach\videomach.exe<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Once you put in the file videomach.exe then videomach should show up.<br />
| <br /><br />
|-<br />
| Switch the file type to MP4<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6a986dcfd4344618.png|374x27px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_da932791910b2a04.png|167x188px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d35497678abed954.png|155x364px]]<br />
|-<br />
| <br /><br />
| Polymer/Monomer Photodegradation: TDESMD 91 121 Bond Data<br />
|-<br />
| <br /><br />
|<br />
Bond lengths shown (white lettering) are chosen to see if this degradation simulation relates to the photo cleavage explained in this specified paper.<br />
<br />
Angew. Chem. Int. Ed. 2015, 54, 1159 –1163<br />
<br />
The transition energy for this excitation is 0.24386664.<br />
<br />
From conformation labeled as &quot;DIMER&quot;<br />
<br />
Acknowledgements: Dr. Dmitri Kilin<br />
|-<br />
| '''Obtaining MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| We will be using python scripts that are premade. In PHOTON<br />
| <br /><br />
|-<br />
| go to the TDESMD_###_### directory you want to go to<br />
| <br /><br />
|-<br />
| in here make another directory called massspectrum<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| mkdir massspectrum<br />
| <br /><br />
|-<br />
| cd masssspectrum<br />
| <br /><br />
|-<br />
| cp /Location/movieALL.xyz<br />
| This should be from the TDESMD_###_### you are in.<br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieALL.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Record number of molecules converted.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| python3 script1n.py<br />
| <br /><br />
|-<br />
| enter number of molecules converted<br />
| <br /><br />
|-<br />
| <br /><br />
| a frequency file should be created<br />
|-<br />
| cp frequency.txt frequency1.txt<br />
| <br /><br />
|-<br />
| awk 'ORS=(!(NR%2)?&quot;\n&quot;:&quot; &quot;)' RS=&quot; &quot; frequency.txt &gt; file1.txt<br />
| <br /><br />
|-<br />
| cp rmovingfile.sh<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
|-<br />
| bash rmovingfile.sh<br />
| <br /><br />
|-<br />
| vi file1.txt<br />
| Edit the file by deleting everything that doesn’t relate to the numbers.<br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| before [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b3731e92878413d2.png|289x328px]]<br />
|<br />
after<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c6aa114dd0a5843f.png|108x333px]]<br />
|-<br />
|<br />
IT IS NECESSARY TO MENTION THAT WHEN CREATING THE FRAGMENTS FILE AN EXTRA BLANK LINE WILL BE GENERATED. IT IS ESSENTIAL TO DELETE.<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_2104ec24e9b14aae.png|258x363px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_36d7bfd4ff6b9e13.png|218x371px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp file1.txt fragments<br />
| <br /><br />
|-<br />
| wc fragments<br />
|<br />
record the first number<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
in my case it is 21<br />
|-<br />
| ~/bin/DRESS_FRAGMENT_v2<br />
| <br /><br />
|-<br />
|<br />
21<br />
<br />
100<br />
<br />
300<br />
<br />
.1<br />
<br />
1<br />
<br />
62<br />
<br />
160<br />
<br />
<br /><br />
|<br />
number of states<br />
<br />
Min MASS (can change if desired)<br />
<br />
Max MASS (can change if desired)<br />
<br />
width of the line used<br />
<br />
magnitude (keep at one)<br />
<br />
number of atoms<br />
<br />
number of electrons<br />
|-<br />
| '''Plotting FERWE MASSPECTRUM'''<br />
| <br /><br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| e in the directory of the file you want to plot, in this example we are using MASS_SPEC.norm as our file with the data we want.<br />
|-<br />
|<br />
set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Monomer H2O Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br />To press the right mouse button and move mouse to preferred window size. Then left click<br />
| monomer h20<br />
|-<br />
| ressing p takes you to the previous zoom level, n to the next level, and u unzooms<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| <br /><br />
|-<br />
| '''Obtaining Multiple Cell MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| Make sure TDESMD CONTCAREXICTED and CONTCARGROUND files have been generated. Start in Cori go to Photon<br />
| <br /><br />
|-<br />
|<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test.sh<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ test_o.sh<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ANA_v2.exe<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/input_ANA<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/CONTCAREXICTED and GROUNDFILES<br />
|<br />
Take these files from Cori and put them into photon.<br />
<br />
These two files are needed for running the scripts properly<br />
<br />
These are the “important parts of the code”<br />
<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
input_ANA should have similar format to this<br />
<br />
Where the first 3 numbers are from POSCAR, the last 3 are the number of cells in the x,y,z axes respectively. This files was duplicated across Z axis.<br />
<br />
18.54438999999999<br />
<br />
14.45937999999995<br />
<br />
17.76793999999994<br />
<br />
1<br />
<br />
1<br />
<br />
10<br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
I used input_ANA<br />
|-<br />
|<br />
CHECK to see if OUTPUT has the correct number of atoms<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_4a92367ce10afd50.png|624x64px]]<br />
| Single cell had 620 atoms, I multipled it by 10 since I wanted to duplicate 10 times. If this is good procede to next steps<br />
|-<br />
| vi test.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_19d33c90b0cefb7a.png|278x215px]]<br />
| change the number to later stage in your trajectory(or the number where you think degradation had occurred. For example my total CONTCARGround and excited was about 191 files. I saw degradation occur later in the simulation so I chose the number 190. This changes with the simulations.<br />
|-<br />
| bash test.sh<br />
| this runs the test.sh file<br />
|-<br />
| bash test_o.sh<br />
| <br /><br />
|-<br />
| NOW LOOKING INTO PHOTON<br />
| <br /><br />
|-<br />
| cp all files into photon directory<br />
| <br /><br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieANA.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
Record number of molecules converted.<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| Follow regular mass spec steps from above. Rather than using MovieALL we are using movieANA.xyz.<br />
| <br /><br />
|-<br />
| <br /><br />
|<br />
photon<br />
<br />
/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
<br />
cori<br />
<br />
/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test<br />
|-<br />
| plummer code for renaming files<br />
| /global/project/projectdirs/m1251/vasp/CHEM676/meade/MONOMER_H2O/TDESMD_162_167/massspec/multiplemassspectest/plumoutput<br />
|-<br />
| '''How to read/get the fragments with their xyz coordinates'''<br />
| <br /><br />
|-<br />
|<br />
Go into the directory with the shot files with the multiple cells. Make sure that some files have this comment inside of them by opening a shot file that has name shot##n#.xyz<br />
<br />
The comment should be.<br />
<br />
created by duplicating cells#<br />
| <br /><br />
|-<br />
| grep -E 'created by duplicating cells.* 17.' shot*<br />
|<br />
finds lines that have both “created by duplicating cells” and the number 17 in the same line.<br />
<br />
17 is the mass number we are looking for<br />
<br />
I am unsure what the “.*” does in this code<br />
<br />
make sure there is a space between the * and the 17<br />
|-<br />
| Then open one of the xyz files listed.<br />
| <br /><br />
|-<br />
| Literally drag and copy the file information into a notepad document in your desktop.<br />
| <br /><br />
|-<br />
| more shot291n10.xyz<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_9c48a58b9e24dc27.png|210x268px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_7c3a89906ed25e30.png|180x270px]]<br />
|-<br />
| open the xyz file in Avogadro and save as mol file.<br />
| <br /><br />
|-<br />
| the Avogadro file is correct visualization of the fragments. Mol file will have some defects so you will have to change those.<br />
| <br /><br />
|-<br />
| Making Video from output of VASP<br />
| <br /><br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exemore<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
I used input_ANA<br />
|-<br />
| '''LAAMPS'''<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_796f3aade1e5b306.png|314x150px]]<br />
|<br />
testwater file:<br />
<br />
Epsolon is Inputs from something like DFT<br />
<br />
<br />
Columns:<br />
<br />
Atom ID<br />
<br />
Molecule ID<br />
<br />
Atom Type<br />
<br />
Q - charge<br />
<br />
X<br />
<br />
Y<br />
<br />
Z<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_f474cba5e65bdaab.png|306x66px]]<br />
| This infor can be put in input script if desired.<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_a9802b534de46367.png|624x180px]]<br />
|<br />
bond id<br />
<br />
bond tuype<br />
<br />
atom id that connects<br />
<br />
atom id that connects as well<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3b00d172c3a15f2b.png|310x103px]]<br />
|<br />
Angle ID<br />
<br />
Angle Type 1 being single bond 2 being double, etc<br />
<br />
Item ID such as first atom O , 2<sup>nd</sup> and 3<sup>rd</sup> atom H<br />
|-<br />
| qsub submit.sh<br />
| submits the job using submit.sh<br />
|-<br />
|<span style="background: #101010">qstat -u meade.erickson</span><br />
| shoes me my jobs<br />
|-<br />
| <span style="background: #101010">chmod -777 lmp_mpi_12Dec18</span><br />
| modifies permission for this file<br />
|-<br />
| [meade.erickson@login0002 testwater]$ chmod -777 ~/bin/lmp_mpi_12Dec18<br />
| giving permission while in a different<br />
|-<br />
| topo readlammpsdata project.data<br />
| reads data file for vmd tk console<br />
|-<br />
| Inputting files, put in .dcd file then right click on it and “Load Data into Molecule” then select .data file.<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_73b2551819922e26.png|287x164px]]<br />
| <br /><br />
|-<br />
| module avail<br />
| Available software in ccast<br />
|-<br />
| qdet job id<br />
| qdel ######<br />
|-<br />
| fragment 0 to 9 or fragment 14<br />
| <br /><br />
|}</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=VASP_and_VMD_on_CCAST&diff=124VASP and VMD on CCAST2022-09-26T21:16:04Z<p>Sysadmin: Sysadmin moved page VASP VMD LAMMPS CCAST LINUX to VASP and VMD on CCAST without leaving a redirect: Page rename</p>
<hr />
<div>__TOC__<br />
<br />
<br /><br />
<br />
<br />
{|<br />
|<br />
'''Login''' '''Photon''':<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
ssh [mailto:kilin@photon.chem.ndsu.nodak.edu <u>kilin@photon.chem.ndsu.nodak.edu</u>]<br />
<br />
cd /scratch001/Chem_476/CHEM676_2019/Meade/DIMER<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
'''Login''' '''Cori''':<br />
<br />
<br /><br />
| Have to be in Photon first then login to Cori<br />
|-<br />
|<br />
ssh -i ~/.ssh/nersc cori.nersc.gov<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
cd /global/cfs/cdirs/m1251/vasp/CHEM676/meade<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
'''Misc. Tools'''<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| cd foldername<br />
| lets you go into a folder<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| cd ..<br />
| Backs you out of folder you are in<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| cd -<br />
| Goes back to previous directory<br />
|-<br />
| cd ~<br />
| Goes to home directory<br />
|-<br />
| freenodes<br />
| in ccast<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| vi filename<br />
| Lets you go into file to edit<br />
|-<br />
| i<br />
| When in “vi” this initiates the editing<br />
|-<br />
| :wq<br />
| This lets you stop editing<br />
|-<br />
| “Ctrl+c” press on keyboard, not a typing thing<br />
| Lets you stop any current running process<br />
|-<br />
| pwd<br />
| Gives you current directory location<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| wc<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| ls<br />
| Lists what is in the current directory<br />
|-<br />
| ls -lt<br />
| Lists what is in the current directory sorting from newest to oldest<br />
|-<br />
| ls –lt W*<br />
| Lists everything that starts with a W sorting from newest to oldest. (Can be used for other letters as well)<br />
|-<br />
| ls -lisanh<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisanh filename<br />
| gives information on the specific file<br />
|-<br />
| ls -lisan<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisan filename<br />
| gives information on the specific file<br />
|-<br />
| ls -a<br />
| shows all files, including hidden ones<br />
|-<br />
| babel –ixyz file.xyz –opdb file.pdb xyz to pdb<br />
|<br />
Using babel, converts the input file of .xyz to outputfile of .pdb<br />
<br />
That is where the –ixyz and –opdb comes from<br />
|-<br />
| “shift+alt+left click”<br />
| For slecting just comlumn or just row information you want exactly<br />
|-<br />
| egrep '163|166' OS_STRENGTH<br />
| grep technique<br />
|-<br />
| more filename<br />
| Lets you see what is in the file<br />
|-<br />
| more ~/bin/filename<br />
| Lets you see the file in a location not in your current directory.<br />
|-<br />
| more filename | grep ### or value | grep again if desired<br />
| shows file and grabs lines with same input you put after the grep command<br />
|-<br />
| cp /Location/filename .<br />
| Copies a file from a different directory location and input it into the current directory. Don’t forget the PERIOD!<br />
|-<br />
| P /Location/filename ./newfilename<br />
| Copies a file from a different directory location and input it into the current directory and rename it in one step<br />
|-<br />
| scp sourceUser:/sourceLocation/filename .<br />
| Takes file from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/* .<br />
| Takes ALL files from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/*.filetype .<br />
| Takes ALL files with that specific file type from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp * destinationUser:/destinationLocation<br />
| Sends all files from source directory to destination directory. Must be in source directory when typing this line<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/DIMabsorptionspec<br />
| Example of scp<br />
|-<br />
| sbatch corifilename<br />
| Lets you submit a file in cori using a corifile made by others<br />
|-<br />
| grep specificdetail filename<br />
| Grabs the specific detail from the file you chose<br />
|-<br />
| sort -n -r filename<br />
| sorts the file in ascending order<br />
|-<br />
| babel BondLength.xyz -O out.xyz --separate<br />
| THIS IS CORRECT WAY THE SEP WORD IS SPELLED INCORRECTLY BUT IS CORRECT FOR BABEL<br />
|-<br />
| awk '{ if($4 &gt;= 4.0000 &amp;&amp; $4&lt;=5.0000) print }' OS_STRENGTH1<br />
| lets you get lines of data that contain the range from 4-5 from column 4<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| users<br />
| lets you see what users are on<br />
|-<br />
| clear<br />
| lets you clear screen<br />
|-<br />
| who<br />
| who is on and more detail about them<br />
|-<br />
| w<br />
| all the people and differing details about them<br />
|-<br />
| id<br />
| lets me see what groups I am in<br />
|-<br />
| freenodes<br />
| lets you see if nodes are available or not<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6335cce9234bc4ef.png|439x225px]]<br />
|<br />
|-<br />
| pbsnodes –avSj | sort -n<br />
| shows available nodes, the | sorts it into order<br />
|-<br />
| pbsnodes –avSj | sort –n | more<br />
| does this in page format rather than all at once<br />
|-<br />
| module avail<br />
| what software is available<br />
|-<br />
| $HOME<br />
| lets you go to your login directory “home” directory<br />
|-<br />
|<br />
thunder.ccast.ndsu.edu<br />
<br />
meade.erickson<br />
| to get into putty<br />
|-<br />
| C:\Users\meade.erickson\Documents\OriginLab\User Files\<br />
| Origin themes location<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Explanations'''<br />
| <br /><br />
|-<br />
| fermi<br />
| fermi energy is referring to the energy difference between the highest and lowest occupied single-particle states. ([https://en.wikipedia.org/wiki/Fermi_energy <u>https://en.wikipedia.org/wiki/Fermi_energy</u>])<br />
|-<br />
| '''Explanations Input Files:'''<br />
| <br /><br />
|-<br />
| INCAR<br />
| It is the central input file of VASP. It determines &quot;what to do and how to do it&quot;. ([https://www.vasp.at/wiki/index.php/INCAR <u>https://www.vasp.at/wiki/index.php/INCAR</u>])<br />
|-<br />
| POSCAR<br />
| <br /><br />
|-<br />
| POTCAR<br />
| <br /><br />
|-<br />
| KPOINTS<br />
| <br /><br />
|-<br />
| '''Explanations Output Files:'''<br />
| <br /><br />
|-<br />
| CONTCAR<br />
|<br />
Contains actual coordinates, velocities, and predictor corrector coordinates. 1<sup>st</sup> block is lattice parameters and atom coordinates. 2<sup>nd</sup> block Initial Velocites for atoms. 3<sup>rd</sup> block Predictor-corrector coordinates.<br />
<br />
Written at end of each job. Used as the POSCAR file for next MD job.<br />
|-<br />
| WAVECAR<br />
| Provides wavefunctions. we usually use it for spectrum data ([https://www.vasp.at/wiki/index.php/WAVECAR <u>https://www.vasp.at/wiki/index.php/WAVECAR</u>])<br />
|-<br />
| OS_STRENGTH<br />
| Information about the OS_STRENGTH<br />
|-<br />
|<br />
<br /><br />
<br /><br />
<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8bf47c7304419724.png|354x154px]]<br />
|<br />
From Wikipedia<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
In spectroscopy, '''oscillator strength''' is a dimensionless quantity that expresses the probability of [https://en.wikipedia.org/wiki/Absorption_(electromagnetic_radiation) <u>absorption</u>] or [https://en.wikipedia.org/wiki/Emission_spectrum <u>emission</u>] of [https://en.wikipedia.org/wiki/Electromagnetic_radiation <u>electromagnetic radiation</u>] in transitions between [https://en.wikipedia.org/wiki/Energy_level <u>energy levels</u>] of an atom or molecule<sup>[</sup>[https://en.wikipedia.org/wiki/Wikipedia:Accuracy_dispute#Disputed_statement <sup>''<u>dubious</u>''</sup>]<sup>'' – ''</sup>[https://en.wikipedia.org/wiki/Talk:Oscillator_strength#Wrong_definition_oscillator_strength <sup>''<u>discuss</u>''</sup>]<sup>]</sup>.<sup><u>[https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Demtr%C3%B6der2003-1 [1]][https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Robinson1996-2 [2]]</u></sup> The oscillator strength can be thought of as the ratio between the quantum mechanical transition rate and the classical absorption/emission rate of a single electron oscillator with the same frequency as the transition.<br />
|-<br />
| POTIM (in INCAR file)<br />
| timestep in femtoseconds. So POTIM=1 is 1 femtosecod<br />
|-<br />
| '''Density of States'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| To get DOS you need to run the correct calculations in VASP using 4 files. INCAR, POSCAR, POTCAR, OUTCAR<br />
| <br /><br />
|-<br />
| Once you get output from your submitted information to cori or photon you can follow these steps to get Density of states<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| Grabs the number of bands (we need this for later) (STATES)<br />
|-<br />
| vi input_overlap<br />
| editing input_overlap<br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| Number of Bands from the grep NBA OUTCAR line<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| quitting editing input_overlap<br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| getting energy pop from different location<br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| wc STATES<br />
| <br /><br />
|-<br />
| grep NELE OUTCAR<br />
| record number of electrons<br />
|-<br />
| grep NIO OUTCAR<br />
| record number of ions<br />
|-<br />
| grep E-fer OUTCAR<br />
| record the three values and their names. If this line doesn’t work the try next line)<br />
|-<br />
| grep ferm OUTCAR | tail –n 1<br />
| record the three values and their names<br />
|-<br />
| ~/bin/DRESS_DOS_NORM<br />
| Executes the DOS code from different file location<br />
|-<br />
|<br />
Number of states from grep NBA OUTCAR<br />
<br />
<br /><br />
| grep NBA OUTCAR line<br />
|-<br />
| -20<br />
| <br /><br />
|-<br />
| 20<br />
| <br /><br />
|-<br />
|<br />
E-fermi info from grep E_fermi<br />
<br />
<br /><br />
|<br />
To literally line up DOS and STATES, you may need to type in 0 for both fermi and alpha+beta.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
A more standard way is to use the middle point energy between HOMO and LUMO i.e. (E_HOMO+E_LUMO)/2 for fermi and 0 for alpha+beta. In this way, the mid point of DOS is at zero, anything below zero is occupied and above zero is unoccupied.<br />
|-<br />
|<br />
alpha+beta info from grep E_fermi<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| .01<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
|<br />
Number of ions from grep NIO OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
Number of electrons from grep NELE OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| 3<br />
| <br /><br />
|-<br />
|<br />
gnuplot ~/bin/gnuprog_DOS.fermi<br />
<br />
<br /><br />
| Plotting the DOS (This is done in NERSC or PHOTON)<br />
|-<br />
|<br />
ps2pdfwr DOS.ps<br />
<br />
<br /><br />
| Converting DOS file from .ps to .pdf<br />
|-<br />
|<br />
*** DISCLAIMER, DON”T HAVE TO USE THISBOX BUT CAN IF DESIRED For better box* this can be one big command or you can split it up into pieces. ( anything before a colon is one command) THIS IS DONE IN PHOTON*<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set out MEADESDOS.ps MAY NOT NEED THIS NOT SURE YET<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
unset key<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set tics font &quot;,30&quot;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set ytics offset 0,0,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set xtics offset -0.4,-1,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
CANT GET THIS TO GO TO OUTPUT FILE NOT SURE WHY<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.518567):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.518567):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.173350):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.17335):4 w l title &quot;Wet Monomer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Wet Monomer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|<br />
I am initially specifying a style that I will be using. In this case chose style 4 and am editing it to what I want.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
lw stands for linwidth, lc line color, not sure where the library is but lc -1 is black, 0=red I think and find out other through trial and error<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
lw is just multiplyer, so lw 4 is 4 times larger than default.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
unset key gets rid of key,<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
[#,#] is the range of my x axis graph.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
($#-###): $# is the column I want to take data from which is being subtracted by the amount of shift I am giving to the data (we want data to be centered at 0 sometimes and so for DOS we take the middle between the start of non occupied plus the end of occupied and divide by 2 to get our shift. (Notoccupied + occupied)/2 ) and the number after the “:” is the column I am correlating the X and Y axis with. So when we plot stuff we need to specify what is the x and what is the y axis.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
not sure exactly what the “w i” portion does but it helps us change from points to lines and then is what fills in the red portion of our graph. (The “i” stands for impulses. Add an extra “lw 2’ to increase the line width as shown in green text.) “w l” does the same thing but DOESN”T fill out the the electron density portion.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
xlabel and y labels are the axis titles and offsets help position them due to the changing sizes of the axes and tics.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Tics are the numbers on the Y and x axis so ytics offsets them and xtics does the horizontal axis change.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Absorption Spectrum - Transition Energy eV – Wavelength nm'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
|<br />
copying INCAR-gm into the directory as long as you are in it<br />
<br />
MAKE SURE<br />
<br />
LWAVE= .TRUE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori07:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec/TEST<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Or make sure the INCAR has this Ag+Carboxyl<br />
<br />
#ISIF=7<br />
<br />
#PREC = HIGH<br />
<br />
ENCUT= 300<br />
<br />
IBRION =3<br />
<br />
POTIM=0.2<br />
<br />
NSW = 30000<br />
<br />
ALGO = NORMAL #48<br />
<br />
ISMEAR = 0<br />
<br />
SIGMA = 0.01<br />
<br />
LREAL = A<br />
<br />
#ISTART = 0<br />
<br />
LWAVE = .TRUE.<br />
<br />
LCHARG = .FALSE.<br />
<br />
EDIFF=1E-6<br />
<br />
EDIFFG=-0.05<br />
<br />
LVTOT = .FALSE.<br />
<br />
#LVDW=.TRUE.<br />
<br />
#SPRING = -5<br />
<br />
#LCLIMB = .TRUE.<br />
<br />
#IMAGES = 7<br />
<br />
#IOPT=7<br />
<br />
#ICHAIN=0<br />
<br />
NELM=40<br />
<br />
#ISPIN=2<br />
<br />
ISYM=0<br />
<br />
NPAR=4<br />
<br />
<br /><br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|345x215px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| It is necessary to have the ‘spectrum’ file made which comes from a healthy WAVECAR which is used to generate OS_STRENGTH. After OS_STRENGHT we make the spectrum ~/bin/SPECTRUM_imp3 file.<br />
| <br /><br />
|-<br />
| Assuming you have conducted a calculation which generated a healthy WAVECAR file.<br />
| <br /><br />
|-<br />
| If healthy WAVECAR exists then…<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| record Number of bands<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
|<br />
1<br />
<br />
Number of Bands<br />
<br />
1<br />
| 36740210<br />
|-<br />
|<br />
‘esc’ key<br />
<br />
:wq<br />
| <br /><br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| <br /><br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| ./osc_str.exe<br />
| This makes the OS_STRENGTH (oscillator strength)<br />
|-<br />
| grep '2.0000 0.0000' OS_STRENGTH &gt; OS_STRENGTHvc<br />
|<br />
Takes the orbital pairs from 2.0000 to 0.0000 into new file.<br />
<br />
<br /><br />
|-<br />
| cp OS_STRENGTH OS_STRENGTH1<br />
| making sure I have original OS_STRENGTH<br />
|-<br />
| cp OS_STRENGTHvc OS_STRENGTH<br />
| making sure I have file OS_STRENGvc kept for 2.0000 – 0.0000 but it will be our OS_STRENGTH<br />
|-<br />
| wc OS_STRENGTH<br />
| Record First number<br />
|-<br />
| ~/bin/SPECTRUM_imp3<br />
| <br /><br />
|-<br />
|<br />
Number from OS_STERNEGTH<br />
<br />
.1<br />
<br />
10<br />
<br />
.001<br />
<br />
HOMO<br />
|<br />
Should be able to find HOMO from STATES file (or energy_pop)<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Can change the “.001” to “0.1 or 0.01” to smooth out the line<br />
|-<br />
| a ‘spectrum’ file should have been made. This holds all the spectrum data. The spectrum can be visualized using these commands in Cori. If you want to visualize in photon then use the gnuplot information below<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE<br />
|<br />
IN CORI YOU CAN USE THESE<br />
<br />
THEN SCP them to photon, then the desktop!<br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE_nm_200_800<br />
|<br />
|-<br />
| ps2pdfwr SPE.ps<br />
|<br />
|-<br />
| ps2pdfwr SPE_nm.ps<br />
|<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec<br />
| example make sure you are in the directory of the files you want to send. Send it to a premade directory in photon<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
FOR PHOTON USE THESE STEPS<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
We are plotting the information of spectrum to get the spectrum in wavelength of nanometers nm and transition energy in electron volts eV<br />
| <br /><br />
|-<br />
| more spectrum<br />
|<br />
(Be sure to be in correct directory)<br />
<br />
Column 1 is the spectrum in wavelength of nm<br />
<br />
column 2 is the spectrum in transition energy of eV<br />
<br />
column 3 is absorbance/absorption in a.u. (absoption units). Absorbance goes with wavelength, absorption goes with transition energy<br />
<br />
column 4 idk what this is.<br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| plot ‘spectrum’ 2:3<br />
| This gives you transition energy in x axis and absorption in a.u for yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [0:10] 'spectrum' using 2:3 w l title &quot;Transition Energy&quot; ls 4; set xlabel 'Transition Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorption a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| This is transition energy in x axis and absorption in yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [200:800] 'spectrum' using 1:3 w l title &quot;Absorption Spectrum&quot; ls 4; set xlabel 'Wavelength, nm' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorbance, a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| nm in x axis and absorbance in yaxis<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''UV-Vis Absorption Spectrum'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
We did one calculation already but we need to do a “geometry optimization” for the next one.<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
| copying INCAR-gm into the directory as long as you are in it<br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|352x219px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| '''Generating CANDIDATES file for Photofragmentation'''<br />
| <br /><br />
|-<br />
| This is after calculations have been ran for MD???<br />
|<br />
Watch this first‼ Part 2 of Dr.Kilins video for how to generate the CANDIDATES file.<br />
<br />
[https://www.youtube.com/watch?v=NIrs1yXJj1Y&feature=youtu.be <u>https://www.youtube.com/watch?v=NIrs1yXJj1Y&amp;feature=youtu.be</u>]<br />
|-<br />
| We want to get information from the WAVECAR that can help us determine the “electron hopping” so we have to find the OS_STRENGTH (oscillator strength) that is high. Since higher OS_Strength the higher the probability that given pair of orbitals is excited. STATES file should have been generated before I think.<br />
|<br />
Watch this second‼ Part 1 of Dr. Kilins video for what to do after CANDIDATES file was generated and explains what is in it.<br />
<br />
[https://www.youtube.com/watch?v=A8BXT-FnuxM&t=212s <u>https://www.youtube.com/watch?v=A8BXT-FnuxM&amp;t=212s</u>]<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| grep –A 700 fermi OUTCAR | tail –n 700 &gt; STATES1<br />
| Fermi information from OUTCAR into a file called STATES 1. Not sure what the –A or 700 mean. But the tail is choosing a specific section of the data, the “tail” end of the chosen information from the OUTCAR file.<br />
|-<br />
| more STATES1<br />
| Determine HOMO and LUMO, HOMO is the highest orbital with a 2, LUMO is the first 0. In my case HOMO is 112 and LUMO is 113<br />
|-<br />
| NOT SURE HOW HE GENERATED STATES FILE<br />
| <br /><br />
|-<br />
| head STATES<br />
| Record the lowest number, in my case it is 81 for DIMER<br />
|-<br />
| tail STATES<br />
| Record the Highest number, in my case it is 160 for DIMER<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| 160<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| more ~/bin/ADVICE.spectrum<br />
| This will make spectrum. This will show 3 lines of code. Literally copy one line and press enter starting with the “module swap PrgEnv-intel PrgEnv-gnu”<br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| copying templates and inputting them into the directory you are in<br />
|-<br />
| ./osc_str_CHEM676.exe<br />
| This reads binary WAVECAR file, Reads input_overlap, creates Oscillator strength<br />
|-<br />
| ls –lt | head<br />
| This should show a fresh OS_STRENGTH was made<br />
|-<br />
| grep ‘2.0000 0.0000’ OS_STRENGTH &gt; OS_NEW<br />
| In the OS_STRENGTH file we will see lines of no transition from occupied to unnoccupied. What we want to see is the lines that have an occupied originally (2.0000) and then are unoccupied (0.0000). This shows excitation. The lines that meet this requirement are then inputted into OS_NEW<br />
|-<br />
| head OS_NEW<br />
| This should show a range of orbitals that have been occupied and are now unoccupied. With the differing orbitals involved. Columns 1 and 2 are the initial and final orbitals that are involved with the “hopping” of electrons. The 3rd column is oscillator Strength. 4th Column is Transition energy.<br />
|-<br />
| mv OS_NEW OS_STRENGTH<br />
| We don’t need all OS_STRENGTH data, just the stuff we “grepped” in OS_NEW. This overrides OS_STRENGTH with information from OS_NEW.<br />
|-<br />
| wc OS_STRENGTH<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| head –n 30 OS_STRENGTH<br />
| In this particular case we only want the top 30 rather than the original value the first column gave was. In my case it was about 1536.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4<br />
| To sort in ascending order of transition energy. 4 is the 4<sup>th</sup> column. I don’t know what –gk means.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4 &gt; CANDIDATES<br />
| Inputs this information into CANDIDATES file<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Orbital Visualization'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| I don’t have all information available here, for example I personally didn’t make the PARCHG files.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
Kilins Guidance to make PARCHG files… I think<br />
<br />
Dear Meade,<br />
<br />
below are steps I did for creating orbitals files on nersc.<br />
<br />
I will let you know when they are ready and instruct on how to visualize.<br />
<br />
slides are good, under given limits.<br />
<br />
In the given time, you may add conclusions as you see them..<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; mkdir ORBITALS<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp WAVECAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp CONTCAR ORBITALS/POSCAR<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp POTCAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp *.sh ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp KPOINTS ORBITALS/<br />
<br />
&gt; cd ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; grep NBA ../OUTCAR<br />
<br />
k-points NKPTS = 1 k-points in BZ NKDIM = 1 number of bands NBANDS= 160<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; cp ~/bin/INCAR/INCAR-pc ./INCAR<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; tail INCAR<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
IBRION=2 #conjugate-gradient algorithm used to relax ions (bad<br />
<br />
NSW=0 #number of ionic steps<br />
<br />
POTIM= .2 #time step in fs<br />
<br />
EDIFFG=-0.001 #minimum energy difference between ionic iterations<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
#EINT= -4.4 -0.5<br />
<br />
EINT= -6 2 <br />
<br />
LVTOT = .TRUE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; sbatch cori_debug.sh <br />
<br />
Submitted batch job 32413471<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
many thanks for the new version!<br />
<br />
The main question in your email is orbitals with VASP:<br />
<br />
a. generate or use premade healthy WAVECAR <br />
<br />
(it is an analog of .chk in gaussian)<br />
<br />
b. in a new directory, use template of the INCAR for orbitals and run vasp once again.<br />
<br />
the templates are named ~/bin/INCAR/INCAR_pc and similar names.<br />
<br />
(it is an analog of the cubegen procedure in gaussian)<br />
<br />
c. upon successful run, there appear files nammed PARCHG.XXXX.ALLK, where XXXX stands for the index of orbital, e.g. PARCHG..01112.ALLK<br />
<br />
(those files are analogs of the .cube files in gaussian. note that vasp DOES not generate literal .cube files!)<br />
<br />
d. download these files and open then in the VMD - the same software you have used for the trajectories.<br />
<br />
I will have a look on the slides and let you nko if any quicl changes are needed or we can go ahead with this version<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
<br /><br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Step 2. Changing PARCHG Files to cube files (For Orbital Visualization)<br />
| From what I have seen we need cube files to show in guassview AND VMD<br />
|-<br />
| Once obtained PARCH.####.ALLK Files then use following command in PHOTON to convert the orbitals you care about to cube!<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| ~/vtstools/chg2cube.pl filename<br />
| <br /><br />
|-<br />
| 3<br />
| selecting PARCHG<br />
|-<br />
| 1 6 7 8<br />
| (These are the atomic numbers for the atoms in your system. In my case I have Hydrogen, Nitrogen, Oxygen, and Carbon<br />
|-<br />
| Your cube file is then generated<br />
| <br /><br />
|-<br />
| SEND cube files, POSCAR (OR CONTCAR), CHGCAR FILES TO DESKTOP<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| Step 3. Using VMD to generate orbitals<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| Open VMD<br />
| <br /><br />
|-<br />
|<br />
In the VMD Main window go to “File” “new Molecule”<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Upload your<br />
<br />
1.POSCAR (OR POSSIBLY CONTCAR): By selecting VASP_POSCAR under the “Determine file type:”<br />
<br />
2. CHGCAR: By selecting VASP_CHGCAR under the “Determine file type:”<br />
<br />
3. USE THIS ONE PARCHG: By selecting VASP_PARCHG under the “Determine file type:” (Even though the file ends in .cube we are classifying it as PARCHG)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b0283f3c67e04e7d.png|219x134px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_eae9e77b87b8627b.png|222x103px]]<br />
|-<br />
|<br />
After all three are uploaded,<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
in the VMD Main window go to “Graphics” and then “Representations”<br />
| <br /><br />
|-<br />
|<br />
In the Graphical Representations window follow these three examples.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
(Change the file you are using by the dropdown menu of “selected Molecule”)<br />
<br />
<br /><br />
|<br />
The red lettering means that PARCHG Isosurface isn’t being shown. You can double click on the red lettering to make it black and showing on the OpenGL Display Window. Double click again to revert to black.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Parchrg can be shown and can be visualized in conjungtion with the CHGCAR file<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_e7a558f4dc169c4e.png|121x260px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b15ffb6b911b765.png|124x266px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3734231f026ab64e.png|124x265px]]<br />
|<br />
|-<br />
|<br />
-You can change the iso value in CHGCAR to get varying orbital visualizations.<br />
<br />
-Play with it.<br />
<br />
-Change the color by changing the ColorID to different ones.<br />
<br />
-(Range in CHGCAR doesn’t matter)<br />
|<br />
For Monomer Project Show<br />
<br />
all index 52 47 29 44 38 28 18 56 64<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
H52<br />
<br />
H47<br />
<br />
H29<br />
<br />
H44<br />
<br />
H38<br />
<br />
H28<br />
<br />
H18<br />
<br />
H56<br />
<br />
H64<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c5f67d67a9a89fa7.png|355x180px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d4c6c68549d8e025.png|357x209px]]<br />
| <br /><br />
|-<br />
| To get rid of opaque water molecuels make both of these in “Graphics” “Materials” 123 is for faded bods and 124 is for the orbitals<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8d905765b5183c3.png|142x229px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_449e5be8bb4a3541.png|139x222px]]<br />
|<br />
|-<br />
| For my Monomer_H2O first frame hide these atoms if you don’t want in view<br />
| <br /><br />
|-<br />
|<br />
<br /><br />
<br /><br />
<br />
<br />
O108<br />
<br />
O99<br />
<br />
O94<br />
<br />
O101<br />
<br />
O81<br />
<br />
O91<br />
<br />
O90<br />
<br />
O86<br />
<br />
O82<br />
<br />
O87<br />
<br />
O79<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
H73<br />
<br />
H45<br />
<br />
H62<br />
<br />
H51<br />
<br />
H58<br />
<br />
H50<br />
<br />
H48<br />
<br />
H14<br />
<br />
H22<br />
<br />
H36<br />
<br />
H42<br />
<br />
H31<br />
<br />
H20<br />
<br />
H23<br />
<br />
H35<br />
<br />
H30<br />
<br />
H54<br />
<br />
H59<br />
<br />
H25<br />
|<br />
all not index ### ### ### ###...<br />
<br />
all not index 108 99 94 101 81 91 90 86 82 87 79 73 45 62 51 58 50 48 14 22 36 42 31 20 23 35 30 54 59 25<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
backbone ithink<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
index 97 103 11 9 10 13 12 96 1 84 78 3 4 100 77 0 8 2 5 6 7<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''These are for the figure first frame for still seeing the water atoms but hazy'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
H22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
O90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
| 22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46 90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Index for Diagram for pics Middle Frame for H20 molecules'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
H63<br />
<br />
H33<br />
<br />
H17<br />
<br />
H21<br />
<br />
60<br />
<br />
68<br />
<br />
42<br />
<br />
38<br />
<br />
34<br />
<br />
50<br />
<br />
72<br />
<br />
73<br />
<br />
16<br />
<br />
61<br />
<br />
55<br />
<br />
71<br />
<br />
59<br />
<br />
69<br />
<br />
40<br />
<br />
14<br />
<br />
39<br />
<br />
74<br />
<br />
67<br />
<br />
53<br />
<br />
49<br />
<br />
41<br />
<br />
52<br />
<br />
27<br />
<br />
15<br />
<br />
31<br />
<br />
23<br />
<br />
54<br />
<br />
62<br />
<br />
32<br />
<br />
64<br />
<br />
30<br />
<br />
26<br />
<br />
57<br />
<br />
37<br />
<br />
51<br />
<br />
58<br />
<br />
70<br />
<br />
43<br />
<br />
25<br />
<br />
35<br />
<br />
46<br />
<br />
45<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|<br />
O80<br />
<br />
87<br />
<br />
99<br />
<br />
81<br />
<br />
88<br />
<br />
105<br />
<br />
104<br />
<br />
91<br />
<br />
90<br />
<br />
106<br />
<br />
97<br />
<br />
108<br />
<br />
102<br />
<br />
89<br />
<br />
93<br />
<br />
101<br />
<br />
92<br />
<br />
107<br />
<br />
78<br />
<br />
109<br />
<br />
86<br />
<br />
83<br />
<br />
94<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
63 33 17 21 60 68 42 38 34 50 72 73 16 61 55 71 59 69 40 14 39 74 67 53 49 41 52 27 15 31 23 54 62 32 64 30 26 57 37 51 58 70 43 25 35 46 45 80 87 99 81 88 105 104 91 90 106 97 108 102 89 93 101 92 107 78 109 86 83 94<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_783e625ce0b8a321.png|182x247px]]<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Preparation of TDESMD Orbital Files and Job Submission'''<br />
| <br /><br />
|-<br />
| mkdir TDESMD_111_114<br />
| We will now use the information of the CANDIDATES file and create new folder for each scenario. You create a file titled using the orbitals involved. In my case the first scenario is with orbitals 111 and 114<br />
|-<br />
| cd TDESMD_111_114<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/INCAR_EXCITED<br />
| Taking INCAR_EXCITED file from location. In my case it is here.<br />
|-<br />
| vi INCAR_EXCITED<br />
| Edit the INCAR_EXCITED file to make it look like<br />
|-<br />
| i<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_79a32df6f9f33bb0.png|303x201px]]<br /><br />
|-<br />
| <br /><br />
|<br />
Change FERWE=#### to match the total number of orbitals/bands. In this example we have a total number of 160 bands/orbitals.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The first segment of FERWE, “110*1” says we have the first 110 orbitals occupied. The second portion is orbital number 111 which we have listed as a “0” signifying this orbital is unnoccopied for our job submission. Then orbital number 112 is occupied with “1”. Then orbital number 113 is unoccupied with “0”. Then orbital number 114 is occupied with “1”. The last part of “46*0” is talking about the other orbitals not mentioned. These orbital should add up to 160. These numbers change with the differing orbitals designating in the CANDIDATES FILE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The main thing that is changing is the position of one occupied orbital and one unoccupied orbital when the compound is in ground state. In this example number 111 is usually occupied in ground state and we are moving the electron to 114 which is unoccupied in ground state.<br />
|-<br />
| <br /><br />
| ONE more example is as follows. With TDESMD_102_115<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_51e84877e4374ca6.png|300x199px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POSCAR ./CONTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/KPOINTS<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/debug1.sh<br />
| <br /><br />
|-<br />
| sbatch debug1.sh<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Creating Videos from movieAll.xyz files and uploading them on youtube'''<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| scp –r TDESMD_###_### DestinationUser:/DestinationLocation<br />
|<br />
Starting in cori<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Using this to send files from cori to photon. We specifically want the movieALL.xyz files.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
This is sending the WHOLE directory and everything it contains<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| mkdir TDESMD_###_###<br />
|<br />
In DESKTOP<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
generate TDESMD_###_### directories<br />
|-<br />
| scp kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_##_###/movieALL.xyz .<br />
| WHILE IN THE TDESMD DIRECTORY THAT YOU WANT TO BE THE DESTINATION take “movieALL.xyz” from photon and put into specific directory<br />
|-<br />
| <br /><br />
| Once the file is in your desktop you can go to your windows folders and open it to check if it is there.<br />
|-<br />
| In windows search bar<br />
| <br /><br />
|-<br />
| VMD<br />
|<br />
type vmd in windows search bar and open it.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
VMD will open with multiple screens.<br />
<br />
<br /><br />
|-<br />
| We will focus on this screen first to open all the extra screens we need. (VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_5f31d31c2c72c623.png|244x114px]]<br />
|-<br />
| Press “File”, “New Molecule,”<br />
| <br /><br />
|-<br />
| On the window that comes up (Molecule File Browser) Press “Browse” and find the file you would like. Find the file you want. In this example it is an xyz file.<br />
| <br /><br />
|-<br />
| Once A file is picked then press “Load” in the Molecule File Browser window.<br />
| <br /><br />
|-<br />
| Going back to VMD Main<br />
| <br /><br />
|-<br />
|<br />
Press “Graphics” and Representations to open a long slender screen.(VMD Main)<br />
<br />
<br /><br />
<br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c8cfeb66ca7723b0.png|225x487px]]<br />
|-<br />
| Press the “Create Rep” Button (Graphical Representation window)<br />
| <br /><br />
|-<br />
| With one of the “Reps” highlighted in green, in the drawing Method switch from Lines to VDW<br />
| <br /><br />
|-<br />
|<br />
Change the Sphere scale to 0.2.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Resolution can stay at default or can be raised if desired.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6ceb4d7e0c7c142a.png|157x340px]]<br />
|-<br />
|<br />
Click on the other line that still says “LINES” and in the Drawing Method change it to Dynamic Bonds<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Change Distance Cutoff to 1.6<br />
<br />
Bond Radius to 0.1<br />
<br />
And Resolution can stay at default or be increased<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_dba6c2b704e3401a.png|158x337px]]<br />
|-<br />
| Go back to VMD Main<br />
| <br /><br />
|-<br />
| Go to “Extensions,” “Visualizations,” and “Movie maker”<br />
| <br /><br />
|-<br />
| Go to the VMD Movie Generator window<br />
| <br /><br />
|-<br />
| In the box of “Name of movie” type the TDESMD_##_### you have.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|214x167px]]<br />
|-<br />
| “Set Working Directory“ choose the destination directory you want the movie to go to.<br />
| <br /><br />
|-<br />
| “Rotation Angle” I put it to 0 but you can put it to whatever.<br />
| <br /><br />
|-<br />
| “trajectory step size” 1<br />
| <br /><br />
|-<br />
| “Movie Duration” This depends on your frames from the (VMDMain) Usually 1000 frames is about 20-30 seconds if the speed (step) is set to 1. But By changing the speed of the video we should make the total video length to 30-45 seconds long.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|232x181px]]<br />
|-<br />
| Back to VMD Main window<br />
| <br /><br />
|-<br />
| Press “Mouse,” “Label,” “Bonds”<br />
| <br /><br />
|-<br />
| click on two neighboring atoms to get the atom name and bond length<br />
| <br /><br />
|-<br />
|<br />
IN VMD Main, go to “Graphics” and “Labels.” select “atoms” in the drop down menu on the “Labels” window.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Select all atoms by clicking on the top atom and then “SHIFT+left click” the bottom atom to select all atoms<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
While all atoms are selected (highlighted in green). Go to “Properties” and in the “Format” box type %e%i.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
(The %e gives the element and the %i gives the index number)<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Some information can be presented by simply having your cursor over the %e%i and a yellow box will pop up for some info.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
You can move the lettering of each atom by moving the offset information.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_46eab8f828aa40bc.png|215x201px]]<br />
|-<br />
| Click the global properties and change text size and text thickness. ( all of this can be see in the (VMD1.9.3 OpenGL Display) [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_260ebe13a2eda05.png|162x114px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3968014bc41ed6a4.png|224x212px]]<br />
|-<br />
| in VMD movie Generator press “make movie” and quickly in VMD main press the play button.<br />
| <br /><br />
|-<br />
|<br />
Once the video is done a popup should be produced asking where to find videomach. Here is the directory for my computer.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
C:\Program Files (x86)\VideoMach\videomach.exe<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Once you put in the file videomach.exe then videomach should show up.<br />
| <br /><br />
|-<br />
| Switch the file type to MP4<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6a986dcfd4344618.png|374x27px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_da932791910b2a04.png|167x188px]]<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d35497678abed954.png|155x364px]]<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| Polymer/Monomer Photodegradation: TDESMD 91 121 Bond Data<br />
|-<br />
| <br /><br />
|<br />
Bond lengths shown (white lettering) are chosen to see if this degradation simulation relates to the photo cleavage explained in this specified paper.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Angew. Chem. Int. Ed. 2015, 54, 1159 –1163<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The transition energy for this excitation is 0.24386664.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
From conformation labeled as &quot;DIMER&quot;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Acknowledgements: Dr. Dmitri Kilin<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Obtaining MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| We will be using python scripts that are premade. In PHOTON<br />
| <br /><br />
|-<br />
| go to the TDESMD_###_### directory you want to go to<br />
| <br /><br />
|-<br />
| in here make another directory called massspectrum<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| mkdir massspectrum<br />
| <br /><br />
|-<br />
| cd masssspectrum<br />
| <br /><br />
|-<br />
| cp /Location/movieALL.xyz<br />
| This should be from the TDESMD_###_### you are in.<br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieALL.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Record number of molecules converted.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| python3 script1n.py<br />
| <br /><br />
|-<br />
| enter number of molecules converted<br />
| <br /><br />
|-<br />
| <br /><br />
| a frequency file should be created<br />
|-<br />
| cp frequency.txt frequency1.txt<br />
| <br /><br />
|-<br />
| awk 'ORS=(!(NR%2)?&quot;\n&quot;:&quot; &quot;)' RS=&quot; &quot; frequency.txt &gt; file1.txt<br />
| <br /><br />
|-<br />
| cp rmovingfile.sh<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
|-<br />
| bash rmovingfile.sh<br />
| <br /><br />
|-<br />
| vi file1.txt<br />
| Edit the file by deleting everything that doesn’t relate to the numbers.<br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| before [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b3731e92878413d2.png|289x328px]]<br />
|<br />
after<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c6aa114dd0a5843f.png|108x333px]]<br />
|-<br />
|<br />
IT IS NECESSARY TO MENTION THAT WHEN CREATING THE FRAGMENTS FILE AN EXTRA BLANK LINE WILL BE GENERATED. IT IS ESSENTIAL TO DELETE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_2104ec24e9b14aae.png|258x363px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_36d7bfd4ff6b9e13.png|218x371px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp file1.txt fragments<br />
| <br /><br />
|-<br />
| wc fragments<br />
|<br />
record the first number<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
in my case it is 21<br />
|-<br />
| ~/bin/DRESS_FRAGMENT_v2<br />
| <br /><br />
|-<br />
|<br />
21<br />
<br />
100<br />
<br />
300<br />
<br />
.1<br />
<br />
1<br />
<br />
62<br />
<br />
160<br />
<br />
<br /><br />
|<br />
number of states<br />
<br />
Min MASS (can change if desired)<br />
<br />
Max MASS (can change if desired)<br />
<br />
width of the line used<br />
<br />
magnitude (keep at one)<br />
<br />
number of atoms<br />
<br />
number of electrons<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Plotting FERWE MASSPECTRUM'''<br />
| <br /><br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| e in the directory of the file you want to plot, in this example we are using MASS_SPEC.norm as our file with the data we want.<br />
|-<br />
|<br />
set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Monomer H2O Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
To press the right mouse button and move mouse to preferred window size. Then left click<br />
| monomer h20<br />
|-<br />
| ressing p takes you to the previous zoom level, n to the next level, and u unzooms<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| '''Obtaining Multiple Cell MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| Make sure TDESMD CONTCAREXICTED and CONTCARGROUND files have been generated. Start in Cori go to Photon<br />
| <br /><br />
|-<br />
|<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test.sh<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ test_o.sh<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ANA_v2.exe<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/input_ANA<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/CONTCAREXICTED and GROUNDFILES<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|<br />
Take these files from Cori and put them into photon.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
These two files are needed for running the scripts properly<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
These are the “important parts of the code”<br />
<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
input_ANA should have similar format to this<br />
<br />
Where the first 3 numbers are from POSCAR, the last 3 are the number of cells in the x,y,z axes respectively. This files was duplicated across Z axis.<br />
<br />
18.54438999999999<br />
<br />
14.45937999999995<br />
<br />
17.76793999999994<br />
<br />
1<br />
<br />
1<br />
<br />
10<br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
I used input_ANA<br />
|-<br />
|<br />
CHECK to see if OUTPUT has the correct number of atoms<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_4a92367ce10afd50.png|624x64px]]<br />
| Single cell had 620 atoms, I multipled it by 10 since I wanted to duplicate 10 times. If this is good procede to next steps<br />
|-<br />
| vi test.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_19d33c90b0cefb7a.png|278x215px]]<br />
| change the number to later stage in your trajectory(or the number where you think degradation had occurred. For example my total CONTCARGround and excited was about 191 files. I saw degradation occur later in the simulation so I chose the number 190. This changes with the simulations.<br />
|-<br />
| bash test.sh<br />
| this runs the test.sh file<br />
|-<br />
| bash test_o.sh<br />
| <br /><br />
|-<br />
| NOW LOOKING INTO PHOTON<br />
| <br /><br />
|-<br />
| cp all files into photon directory<br />
| <br /><br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieANA.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Record number of molecules converted.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| Follow regular mass spec steps from above. Rather than using MovieALL we are using movieANA.xyz.<br />
| <br /><br />
|-<br />
| <br /><br />
|<br />
photon<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cori<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| plummer code for renaming files<br />
| /global/project/projectdirs/m1251/vasp/CHEM676/meade/MONOMER_H2O/TDESMD_162_167/massspec/multiplemassspectest/plumoutput<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''How to read/get the fragments with their xyz coordinates'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
Go into the directory with the shot files with the multiple cells. Make sure that some files have this comment inside of them by opening a shot file that has name shot##n#.xyz<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The comment should be.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
created by duplicating cells#<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| grep -E 'created by duplicating cells.* 17.' shot*<br />
|<br />
finds lines that have both “created by duplicating cells” and the number 17 in the same line.<br />
<br />
17 is the mass number we are looking for<br />
<br />
I am unsure what the “.*” does in this code<br />
<br />
make sure there is a space between the * and the 17<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Then open one of the xyz files listed.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Literally drag and copy the file information into a notepad document in your desktop.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| more shot291n10.xyz<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_9c48a58b9e24dc27.png|210x268px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_7c3a89906ed25e30.png|180x270px]]<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| open the xyz file in Avogadro and save as mol file.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| the Avogadro file is correct visualization of the fragments. Mol file will have some defects so you will have to change those.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| Making Video from output of VASP<br />
| <br /><br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exemore<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
I used input_ANA<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| '''LAAMPS'''<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_796f3aade1e5b306.png|314x150px]]<br />
|<br />
testwater file:<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Epsolon is Inputs from something like DFT<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Coloumns:<br />
<br />
Atom ID<br />
<br />
Molecule ID<br />
<br />
Atom Type<br />
<br />
Q - charge<br />
<br />
X<br />
<br />
Y<br />
<br />
Z<br />
<br />
<br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_f474cba5e65bdaab.png|306x66px]]<br />
| This infor can be put in input script if desired.<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_a9802b534de46367.png|624x180px]]<br />
|<br />
bond id<br />
<br />
bond tuype<br />
<br />
atom id that connects<br />
<br />
atom id that connects as well<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3b00d172c3a15f2b.png|310x103px]]<br />
|<br />
Angle ID<br />
<br />
Angle Type 1 being single bond 2 being double, etc<br />
<br />
Item ID such as first atom O , 2<sup>nd</sup> and 3<sup>rd</sup> atom H<br />
|-<br />
| qsub submit.sh<br />
| submits the job using submit.sh<br />
|-<br />
| <span style="background: #101010">qstat -u meade.erickson</span><br />
| shoes me my jobs<br />
|-<br />
| <span style="background: #101010">chmod -777 lmp_mpi_12Dec18</span><br />
| modifies permission for this file<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| [meade.erickson@login0002 testwater]$ chmod -777 ~/bin/lmp_mpi_12Dec18<br />
| giving permission while in a different<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| topo readlammpsdata project.data<br />
| reads data file for vmd tk console<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Inputting files, put in .dcd file then right click on it and “Load Data into Molecule” then select .data file.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_73b2551819922e26.png|287x164px]]<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| module avail<br />
| Available software in ccast<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| qdet job id<br />
| qdel ######<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| fragment 0 to 9 or fragment 14<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|}<br />
<br />
<br /><br />
<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=TeraTermInsteadofHyperTerminal&diff=123TeraTermInsteadofHyperTerminal2022-09-26T21:15:36Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== Open TeraTerm ======<br />
* Select the “Serial”<br />
* Select COM1<br />
* (If it doesn’t connect you may need to select other COM)<br />
* Go to “File”<br />
* Select “Log..”<br />
* Type/Select the “.txt” file you would like to save the data to. THE FILE NEEDS TO HAVE “.txt” EXTENSION ADDED TO IT IF IT DOESN’T ALREADY HAVE IT<br />
* Once selected a second popup window named “Tera Term: Log” will show during the logging process. (No need to touch this window yet.)<br />
* Go to the Tera Term Command window and Type “GA”<br />
* The terminal will retrieve the data from the specific you are on in the BODTrak II machine.<br />
* Go to the secondary popup window and press “Close” when complete.<br />
* Repeat this process for all 6 channels of the BODTRAK II Machine.<span id="_GoBack"></span></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Generating_Descriptors_Workflow&diff=122Generating Descriptors Workflow2022-09-26T21:14:49Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
Procedure Document for using Chem Sketch, Avogadro, Dragon, Excel, Hyperchem, &amp; Build QSAR to obtain Descriptors.<br />
<br />
Purpose: Using ChemSketch, ChemDraw, HyperChem, Avogadro, Dragon5, Build QSAR, and QSARINS programs to generate structures, descriptors, and models. OpenBabel is used for converting files from one type to another.<br />
<br />
[[File:GeneratingDescriptors_Model_ChemSketch_HyperChem_OpenBabel_Avogadro_Dragon_BuildQSAR_HTML_e43805e4b85a54a7.png|624x291px]]<br />
<br />
====== ChemSketch ======<br />
Can convert SMILES to structure and structure to SMILES.<br />
<br />
# Open Chem Sketch and click out of the little popup windows by clicking the X on the top right of the small window.<br />
# Draw the desired molecule in skeletal format.<br />
# Once the desired molecule is drawn then go to “Tools” and click “Clean Structure.” This will make it easier for the other programs to read what you created.<br />
# Next, go to “Tools” again and then “Generate.” You will have to generate the “Name of Structure,” “Smiles Notation,” and “InChi for Structure.”<br />
# Open a blank document on Excel. For the molecule created copy and paste the three things you generated. These three things will help keep your data organized. You may not need to refer to all three of these again but it is best to have them ready on hand as a precaution. You will use the “Name of Structure” for naming the files. ** Make sure you assign a number to each molecule. If you have 10 molecules just assign them numbers 1-10. Once again this will help with organizing your data. Organization is a big thing in this work!<br />
# After the three things you generated are on Excel and you assigned the molecule a number. Then delete the names off of chem sketch. Keep the structure and save it as an “MDL Molfiles (*.mol).” The file name can be whatever but it is preferred to follow this format. “Assigned # Name of Structure” Example… “33 2-ethenyl-5-nitrofuran.mol” The “.mol” should be automatically added once you select “MDL molfiles (*.mol).<br />
# Exit Chem Sketch.<br />
<br />
====== ChemDraw ======<br />
SPECIAL can convert name of structure to a structure.<br />
# ChemDraw has a special feature that if given a name of a polymer, it will generate the monomer unit for you.<br />
# Open ChemDraw Professional<br />
# Go to “Structure”<br />
# Select “Convert Name to Structure”<br />
# Type in the name of the polymer without “poly” for example. “Polybutane” only type “butane.”<br />
# It will give you a generalized version. This will assist in giving you the correct monomer unit or just generalized version.<br />
# *** ONLY HAVE STURCTURE IN “*.MOL” FILE*** Delete the text after the model is generated because the text will cause issues later<br />
# NEXT STEPS ARE FOR CREATION OF STRUCTURE AND NAME REGULAR WAY LIKE DRAWING<br />
# Draw the desired molecule<br />
# Select the model by pressing the logo near the top left of the screen [[File:GeneratingDescriptors_Model_ChemSketch_HyperChem_OpenBabel_Avogadro_Dragon_BuildQSAR_HTML_c603a81bb8df177f.png|22x25px]]<br />
# Once selected go to “Structure” and click “Clean up Structure.” This will make it easier for the other programs to read what you created.<br />
# To generate the name of this structure go to “Structure” and press “Convert structure to name.”<br />
# Open a blank document on Excel. For the molecule created copy and paste the name you generated. This will help keep your data organized.<br />
# After the model is generated and the name is on Excel, assign a number to the molecule on excel. Then delete the names off of ChemDraw. Keep the structure and save it as an “MDL Molfiles (*.mol).” The file name can be whatever but it is preferred to follow this format. “Assigned # Name of Structure” Example… “33 2-ethenyl-5-nitrofuran.mol” The “.mol” should be automatically added once you select “MDL molfiles (*.mol).<br />
# *** ONLY HAVE STURCTURE IN “*.MOL” FILE*** Delete the text after the model is generated because the text will cause issues later<br />
# Exit Chem Draw.<br />
<br />
====== Avogadro ======<br />
<ol start="17"><br />
<li><p>After doing these steps for all the molecules you wanted to create then open the program Avogadro.</p></li><br />
<li><p>Go to “File” and “Open” to bring a molecule you created in ChemSketch to Avogadro. Find your molecule that you want to bring to Avogadro and select it to open it. Click “Yes” when a small window pops up about 3D coordinates and a rough sketch.</p></li><br />
<li><p>Once the molecule is on Avogadro then go to “Extensions” and select “Optimize geometry.”</p></li><br />
<li><p>* There is another way to Optimize Geometry How do we do it again???</p></li><br />
<li><p>Save the file as “Sybyl Mol2 (*.mol2)”</p></li><br />
<li><p>Exit Avogadro.</p></li></ol><br />
<br />
====== Dragon5 ======<br />
It is okay to use *.mol and *.hin or any other format usable on Dragon, no issues what so ever)<br />
<br />
<ol start="23"><br />
<li><p>Open “Dragon5.exe,” exit out of the small windows that pup up.</p></li><br />
<li><p>Select “Calculate Descriptors” and then select all the “.mol2” files that you want to calculate. Use the ‘Ctrl’ button when selecting the files so all desired files can be selected.</p></li><br />
<li><p>Press the green check mark “OK” when all desired files are selected.</p></li><br />
<li><p>Choose the desired descriptors you want calculated for these molecule files. “X” means checked. Then press RUN.</p></li><br />
<li><p>Press Continue when the small window pops up.</p></li><br />
<li><p>A yellow window pops up and gives information about the calculations. If no errors are listed on the yellow window then exit out of it.</p></li><br />
<li><p>Select “Save Descriptors.”</p></li><br />
<li><p>Make sure “Constant Variables” &amp; “Near-Constant Variables” are selected “x.” Select “Pair Correlation” and pick .95 or something around that number.</p></li><br />
<li><p>Press “Save” and save as “.txt” file.</p></li><br />
<li><p>Leave Dragon open for future use.</p></li><br />
<li><p>Find the file you saved and open with Notepad++. You can do this by finding the file and right clicking on the file. Select “Edit with Notepad++” to open it in Notepad++.</p></li><br />
<li><p>In Notepad++, select all the information and copy onto an EXCEL worksheet. If desired you can delete the first 2 rows of the information on Notepad++. It is recommended to write the number of descriptors. This will help you later. It is listed in the 2<sup>nd</sup> row, 3<sup>rd</sup> column.</p></li><br />
<li><p>Once all the information is saved onto Excel then exit out of the Notepad++ file.</p></li></ol><br />
<br />
====== Splitting to Training and Test sets ======<br />
In Excel worksheet, split the whole set to training and test sets by sorting your molecules according to “Experimental values” or “Y” column, in descending or ascending order. After sorting, select every 5<sup>th</sup> (20% of the set) or every 4<sup>th</sup> (25% of the set) compound (row) and then copy it to another excel sheet, naming it as a Test set. Remove the test compounds from the initial set to make a training set.<br />
<br />
Use the obtained training set to build a dataset in BuildQSAR.<br />
<br />
====== BuildQSAR ======<br />
<ol start="36"><br />
<li><p>Open “BuildQSAR.”</p></li><br />
<li><p>Go to “File” then “New.”</p></li><br />
<li><p>Add a Title to “Dataset Title.”</p></li><br />
<li><p>Change the number of “Compounds” to the amount in your data. Change the number of “Descriptors” you have. The number of descriptors was obtained from the Notepad++ information.</p></li><br />
<li><p>Click “Ok.”</p></li><br />
<li><p>Input observed data into the yellow column. The yellow column is for the observed information from other sources such as research papers.</p></li><br />
<li><p>** The observed data should be put into Excel at this point.</p></li><br />
<li><p>Copy and paste the descriptor information from Excel into the blue cells of BuildQSAR.</p></li><br />
<li><p>Go to “QSAR” then “Variable Selection” then “Systematic Search” or “Genetic Algorithm.” (note: Choose Genetic Algorithm only when you need 4, 5 or higher number of variables in the model).</p></li><br />
<li><p>A small popup window will pop up. Make sure the 2 boxes under “Cross Validation” are checked. The correlation criteria can change but if uncertain on a number then put 0.6 as default.</p></li><br />
<li><p>“Descriptors per Model,” this is usually calculated using the 5-1 rule. The 5-1 rule relates the number of molecules you have to the number “Variables AKA Descriptors” in your “Model oKA Equation.” Example: 5-1 rule is used on 24 molecules you should have 4 in the “Descriptors per model” section. ** DON’T ROUND UP **</p></li><br />
<li><p>“No. of generations” can vary 200-500), but 200 is an okay default number to have.</p></li><br />
<li><p>“Models per Generation” should be at least 3 (better to have between 5-10).</p></li><br />
<li><p>Press “Run.”</p></li><br />
<li><p>Double Click on any of the cells in the first row.</p></li><br />
<li><p>A pop up window with a “Model aka equation” will pop up.</p></li><br />
<li><p>Copy and paste the model and information in the “()” onto Excel.</p></li><br />
<li><p>Close out of the window with the model information.</p></li><br />
<li><p>Copy the model and () information from all three rows.</p></li><br />
<li><p>These rows are the different models the BuildQSAR generated for you.</p></li></ol><br />
<br />
====== Descriptor Information ======<br />
<ol start="56"><br />
<li><p>So each model (AKA equation) has a certain number of descriptors in it. In this example there are 3.<math display="block">Y_1 = -1.6198 (\pm 1.1586) X_{269} + 0.0110 (\pm 0.0016) X_{631} - 0.0336 (\pm 0.0094) X_{634} + 0.8483 (\pm 0.3386)</math></p><br />
</li><br />
<li><p>To find the meaning of these descriptors go to your Excel file and above your descriptor data create a row that is listed from X1 to X#. X1 is above the first descriptor, usually MW. The #’s are the number of descriptors in one row.</p></li><br />
<li><p>On Excel find the descriptors that are listed in the model. Find the abbreviation under the X value. Example X269 corresponded to the abbreviation “MATS3e”</p></li><br />
<li><p>Once you have all the descriptor abbreviations then go back to Dragon.</p></li><br />
<li><p>Select “ Descriptor Search”</p></li><br />
<li><p>Type an abbreviation you found from the Excel file.</p></li><br />
<li><p>Press “Search”</p></li><br />
<li><p>This will give the descriptor information.</p></li></ol><br />
<br />
====== Test Set predictions ======<br />
Use obtained model to calculate “Predicted Experimental values” for the Test set.<br />
<br />
For that, find the descriptors selected by the model, find them among test set descriptors, and then calculate “Predicted Experimental values” by applying mathematical model from training set and values of selected descriptor for each compound in the test set.<br />
<br />
Do the same for all models, from 1 to 5 variables. Calculate predictive R2 values by correlation of “Experimental values” and Predicted Experimental values” in Excel (function CORR( )).<br />
<br />
Build a graph for all models from 1 to 5 variables, by drawing on axes X No of model and on axes Y the values R<sup>2</sup> for each model (separate lines for training and for the tests sets).<br />
<br />
Select the best model by finding the maximum value of R<sup>2</sup> for the test set models.<br />
<br />
====== Open Babel GUI ======<br />
This section depends on conversion of one file type to another if needed. Usually this is done to get HOMO LUMO (Extra descriptors) from Hyperchem (another 3D software) which has file types of “*.hin” Also, this is used to convert file type to smiles notation if needed. In this example we are converting “*.mol” files to Smiles format notation and “*.hin” format. “*.hin” is for HyperChem<br /><br />
<ol start="64"><br />
<li><p>Open “Open Babel GUI”</p></li><br />
<li><p>On the top left of the window that opens chose the file type that you have. For example “mol—MDL MOL format is a common file extension.”</p></li><br />
<li><p>Once chosen you can select the way you want the data read in OpenBabel. In this example the “Use this format for all input files (ignore file extensions)” will be checked.</p></li><br />
<li><p>Clicking on the left side “…” box you can choose the file you would like converted.</p></li><br />
<li><p>Once the file was chosen it should be in the bar to the left of the “…” box</p></li><br />
<li><p>Under the very top right comment of “OUTPUT FORMAT” chose what you would like converted to. In this example chose “smiles - - SMILES format”</p></li><br />
<li><p>Check “Add Hydrogens (make explicit) in the middle column</p></li><br />
<li><p>Check “Generate 3D coordinates” in the middle column</p></li><br />
<li><p>Simply press convert and the data you want should be listed in the right column.</p></li><br />
<li><p>Copy the Smiles notation and save it onto another file such as excel or notepad. This is one example of what a smiles notation looks like “OC(F)(F)[C@H](F)C(F)(F)F”</p></li><br />
<li><p>The same step as 62 can be used to create the “*.hin” format notation. This is done by choosing “hin - - HyperChem HIN format” under OUTPUT FORMAT.</p></li><br />
<li><p>Make sure “Output below only (no output file)” and “Display in firefox” are NOT checked. These are on the right side column or AKA output column.</p></li><br />
<li><p>Before pressing Convert you need to specify an output file and location. A good idea is to name it exactly the same as the input fil except change the “.mol” ending file to “.hin” (THE “.hin” NEEDS TO BE INCLUDED IN THE OUTPUT FILE)</p></li><br />
<li><p>Press convert</p></li><br />
<li><p>The output file should be in the location with the name specified.</p></li></ol><br />
<br />
====== HyperChem ======<br />
This section depends on the need for quantum descriptors such as HOMO, LUMO, and Total energy.<br />
<br />
<ol start="79"><br />
<li><p>Open “HyperChem”</p></li><br />
<li><p>Open a file that has “.hin” extension</p></li><br />
<li><p>Press “Setup” on the tool bar</p></li><br />
<li><p>Press “Semi-empirical”</p></li><br />
<li><p>Press “RM1”</p></li><br />
<li><p>Press “OK</p></li><br />
<li><p>Press “Compute” on the tool bar</p></li><br />
<li><p>Select “Geometry Optimization”</p></li><br />
<li><p>For regular work make sure “Polak –Ribiere (Conjugate gradiant)” is selected.</p></li><br />
<li><p>Have 0.1 in the top bar of RMS gradiant and in the bottom bar 600.</p></li><br />
<li><p>Have “In vacuo” selected</p></li><br />
<li><p>For the “Screen refresh period” use 1 for cycles.</p></li><br />
<li><p>Steps 78-81 should be default settings</p></li><br />
<li><p>Press “OK”</p></li><br />
<li><p>Once the optimization has stopped (Numbers on the bottom right or left have stopped moving) then select “Compute”</p></li><br />
<li><p>Select “Properties”</p></li><br />
<li><p>All the information in “Properties” can be used as quantum descriptors if needed</p></li><br />
<li><p>By Pressing “Details” you will get more information and these are also used as descriptors if needed.</p></li><br />
<li><p>Exit out of “Properties”</p></li><br />
<li><p>Press “Compute”</p></li><br />
<li><p>Select “Orbitals”</p></li><br />
<li><p>the screen that pops up shows 2-4 different columns, the left being Alpha orbitals and the right being Beta orbitals. The pink/purple column is simply the orbitals above the HOMO-LUMO Gap, and the green is simply below the HOMO LUMO Gap</p></li><br />
<li><p>Each line in these columns gives you the energy in “eV” for that orbital.</p></li><br />
<li><p>If you zoom into the bottom most line of the pink/purple column that is LUMO- Lowest unoccupied molecular orbital. If you select the bottom most line of the pink/purple column, it will give you a value in the “Energy” bar as “eV”. That is LUMO.</p></li><br />
<li><p>If you zoom into the top most line of the green column that is HOMO- Highest Occupied Molecular Orbital. If you select the top most like of the green column it will give you a value in the “Energy” bar as “eV”. That is HOMO</p></li><br />
<li><p>The difference of these energies is the HOMO-LUMO Gap.</p></li><br />
<li><p>If desired, you can simply “Plot” isosurfaces of the orbital that is selected by pressing the “Plot” button on the bottom left of the Orbitals panal.</p></li><br />
<li><p>Press “OK” when done with this screen.</p></li><br />
<li><p>To get further Descriptors</p></li><br />
<li><p>Select “Compute”</p></li><br />
<li><p>Select “ QSAR Properties” at the bottom</p></li><br />
<li><p>In this window you can select any of the properties that you desired to compute by simply selecting the property and press compute. The value will be shown at the bottom of the “QSAR Properties” window.</p></li><br />
<li><p>Scripts are also available to be written for Hyperchem if desired, you can do this via following Lesson 17 or 18 in this manual. <u>http://www.chemistry-software.com/pdf/Hyperchem_full_manual.pdf</u><br /></p><br />
<li><p>If manual doesn’t show up just google “Hyperchem Manual”</p></li></li><br />
<li><p>If It says your Molecule has issues before Geometry optimization such as Valence error go to Select</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>Select All</p></li><br />
<li><p>Select again</p></li><br />
<li><p>“Add H &amp; Model Build”</p></li></ol><br />
</li><br />
<li><p>This corrects the error</p></li><br />
<li><p>Additionally, If molecules are large and “Semi-Empirical” geometry optimization takes too much time then molecular mechanics is needed</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>When doing the molecular mechanics it is necessary to make sure the atom “type” is corrected.</p></li><br />
<li><p>To check if atom type is correct go to “Display”</p><br />
<ol style="list-style-type: lower-roman;"><br />
<li><p>“Labels”</p></li><br />
<li><p>“type”</p></li></ol><br />
</li><br />
<li><p>If you see “**” on the atoms then hyperchem needs to recalculate the atoms to get C, H, O, and other atoms.</p><br />
<ol style="list-style-type: lower-roman;"><br />
<li><p>Go to “setup”</p></li><br />
<li><p>“Molecular Mechanics”</p></li><br />
<li><p>Select “MM+”</p></li><br />
<li><p>Click on “Options”</p></li><br />
<li><p>Press OK</p></li><br />
<li><p>Click on “Components”</p></li><br />
<li><p>Press OK</p></li><br />
<li><p>Then press “OK” on the same window you selected “MM+”</p></li><br />
<li><p>You should get a popup window saying the atom types are going to be recalculated</p></li><br />
<li><p>Press “OK”</p></li></ol><br />
</li><br />
<li><p>This should change the “**” to letters such as C, H, O</p></li><br />
<li><p>If it doesn’t initially then try going to labels and pressing “none.” Then repeat steps 114 a-b</p></li><br />
<li><p>If that doesn’t work then select the semi-empirical. Once selected redo steps 114a-d</p></li><br />
<li><p>Use RHF not UHF</p></li></ol><br />
</li></ol><span id="_GoBack"></span></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Generating_Descriptors_Workflow&diff=121Generating Descriptors Workflow2022-09-26T21:03:27Z<p>Sysadmin: Sysadmin moved page GeneratingDescriptors Model ChemSketch HyperChem OpenBabel Avogadro Dragon BuildQSAR to Generating Descriptors Workflow without leaving a redirect: Page rename</p>
<hr />
<div>__TOC__<br />
<br />
Procedure Document for using Chem Sketch, Avogadro, Dragon, Excel, Hyperchem, &amp; Build QSAR to obtain Descriptors.<br />
<br />
Purpose: Using ChemSketch, ChemDraw, HyperChem, Avogadro, Dragon5, Build QSAR, and QSARINS programs to generate structures, descriptors, and models. Also, OpenBabel is used for converting files from one type to another.<br />
<br />
[[File:GeneratingDescriptors_Model_ChemSketch_HyperChem_OpenBabel_Avogadro_Dragon_BuildQSAR_HTML_e43805e4b85a54a7.png|624x291px]]<br />
<br />
Chem Sketch: Special can convert SMILES to structure and structure to SMILES.<br />
<br />
# Open Chem Sketch and click out of the little popup windows by clicking the X on the top right of the small window.<br />
# Draw the desired molecule in skeletal format.<br />
# Once the desired molecule is drawn then go to “Tools” and click “Clean Structure.” This will make it easier for the other programs to read what you created.<br />
# Next, go to “Tools” again and then “Generate.” You will have to generate the “Name of Structure,” “Smiles Notation,” and “InChi for Structure.”<br />
# Open a blank document on Excel. For the molecule created copy and paste the three things you generated. These three things will help keep your data organized. You may not need to refer to all three of these again but it is best to have them ready on hand as a precaution. You will use the “Name of Structure” for naming the files. ** Make sure you assign a number to each molecule. If you have 10 molecules just assign them numbers 1-10. Once again this will help with organizing your data. Organization is a big thing in this work!<br />
# After the three things you generated are on Excel and you assigned the molecule a number. Then delete the names off of chem sketch. Keep the structure and save it as an “MDL Molfiles (*.mol).” The file name can be whatever but it is preferred to follow this format. “Assigned # Name of Structure” Example… “33 2-ethenyl-5-nitrofuran.mol” The “.mol” should be automatically added once you select “MDL molfiles (*.mol).<br />
# Exit Chem Sketch.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Chem Draw: SPECIAL can convert name of structure to a structure.<br />
<br />
# ChemDraw has a special feature that if given a name of a polymer, it will generate the monomer unit for you.<br />
# Open ChemDraw Professional<br />
# Go to “Structure”<br />
# Select “Convert Name to Structure”<br />
# Type in the name of the polymer without “poly” for example. “Polybutane” only type “butane.”<br />
# It will give you a generalized version. This will assist in giving you the correct monomer unit or just generalized version.<br />
# *** ONLY HAVE STURCTURE IN “*.MOL” FILE*** Delete the text after the model is generated because the text will cause issues later<br />
# NEXT STEPS ARE FOR CREATION OF STRUCTURE AND NAME REGULAR WAY LIKE DRAWING<br />
# Draw the desired molecule<br />
# Select the model by pressing the logo near the top left of the screen [[File:GeneratingDescriptors_Model_ChemSketch_HyperChem_OpenBabel_Avogadro_Dragon_BuildQSAR_HTML_c603a81bb8df177f.png|22x25px]]<br />
# Once selected go to “Structure” and click “Clean up Structure.” This will make it easier for the other programs to read what you created.<br />
# To generate the name of this structure go to “Structure” and press “Convert structure to name.”<br />
# Open a blank document on Excel. For the molecule created copy and paste the name you generated. This will help keep your data organized.<br />
# After the model is generated and the name is on Excel, assign a number to the molecule on excel. Then delete the names off of ChemDraw. Keep the structure and save it as an “MDL Molfiles (*.mol).” The file name can be whatever but it is preferred to follow this format. “Assigned # Name of Structure” Example… “33 2-ethenyl-5-nitrofuran.mol” The “.mol” should be automatically added once you select “MDL molfiles (*.mol).<br />
# *** ONLY HAVE STURCTURE IN “*.MOL” FILE*** Delete the text after the model is generated because the text will cause issues later<br />
# Exit Chem Draw.<br />
<br />
Avogadro:<br />
<br />
<ol start="17"><br />
<li><p>After doing these steps for all the molecules you wanted to create then open the program Avogadro.</p></li><br />
<li><p>Go to “File” and “Open” to bring a molecule you created in ChemSketch to Avogadro. Find your molecule that you want to bring to Avogadro and select it to open it. Click “Yes” when a small window pops up about 3D coordinates and a rough sketch.</p></li><br />
<li><p>Once the molecule is on Avogadro then go to “Extensions” and select “Optimize geometry.”</p></li><br />
<li><p>* There is another way to Optimize Geometry How do we do it again???</p></li><br />
<li><p>Save the file as “Sybyl Mol2 (*.mol2)”</p></li><br />
<li><p>Exit Avogadro.</p></li></ol><br />
<br />
Dragon5: (IT IS OKAY TO USE *.mol and *.hin or any other format usable on Dragon, no issues what so ever)<br />
<br />
<ol start="23"><br />
<li><p>Open “Dragon5.exe,” exit out of the small windows that pup up.</p></li><br />
<li><p>Select “Calculate Descriptors” and then select all the “.mol2” files that you want to calculate. Use the ‘Ctrl’ button when selecting the files so all desired files can be selected.</p></li><br />
<li><p>Press the green check mark “OK” when all desired files are selected.</p></li><br />
<li><p>Choose the desired descriptors you want calculated for these molecule files. “X” means checked. Then press RUN.</p></li><br />
<li><p>Press Continue when the small window pops up.</p></li><br />
<li><p>A yellow window pops up and gives information about the calculations. If no errors are listed on the yellow window then exit out of it.</p></li><br />
<li><p>Select “Save Descriptors.”</p></li><br />
<li><p>Make sure “Constant Variables” &amp; “Near-Constant Variables” are selected “x.” Select “Pair Correlation” and pick .95 or something around that number.</p></li><br />
<li><p>Press “Save” and save as “.txt” file.</p></li><br />
<li><p>Leave Dragon open for future use.</p></li><br />
<li><p>Find the file you saved and open with Notepad++. You can do this by finding the file and right clicking on the file. Select “Edit with Notepad++” to open it in Notepad++.</p></li><br />
<li><p>In Notepad++, select all the information and copy onto an EXCEL worksheet. If desired you can delete the first 2 rows of the information on Notepad++. It is recommended to write the number of descriptors. This will help you later. It is listed in the 2<sup>nd</sup> row, 3<sup>rd</sup> column.</p></li><br />
<li><p>Once all the information is saved onto Excel then exit out of the Notepad++ file.</p></li></ol><br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Splitting to Training and Test sets<br />
<br />
In Excel worksheet, split the whole set to training and test sets by sorting your molecules according to “Experimental values” or “Y” column, in descending or ascending order. After sorting, select every 5<sup>th</sup> (20% of the set) or every 4<sup>th</sup> (25% of the set) compound (row) and then copy it to another excel sheet, naming it as a Test set. Remove the test compounds from the initial set to make a training set.<br />
<br />
Use the obtained training set to build a dataset in BuildQSAR.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
BuildQSAR:<br />
<br />
<ol start="36"><br />
<li><p>Open “BuildQSAR.”</p></li><br />
<li><p>Go to “File” then “New.”</p></li><br />
<li><p>Add a Title to “Dataset Title.”</p></li><br />
<li><p>Change the number of “Compounds” to the amount in your data. Change the number of “Descriptors” you have. The number of descriptors was obtained from the Notepad++ information.</p></li><br />
<li><p>Click “Ok.”</p></li><br />
<li><p>Input observed data into the yellow column. The yellow column is for the observed information from other sources such as research papers.</p></li><br />
<li><p>** The observed data should be put into Excel at this point.</p></li><br />
<li><p>Copy and paste the descriptor information from Excel into the blue cells of BuildQSAR.</p></li><br />
<li><p>Go to “QSAR” then “Variable Selection” then “Systematic Search” or “Genetic Algorithm.” (note: Choose Genetic Algorithm only when you need 4, 5 or higher number of variables in the model).</p></li><br />
<li><p>A small popup window will pop up. Make sure the 2 boxes under “Cross Validation” are checked. The correlation criteria can change but if uncertain on a number then put 0.6 as default.</p></li><br />
<li><p>“Descriptors per Model,” this is usually calculated using the 5-1 rule. The 5-1 rule relates the number of molecules you have to the number “Variables AKA Descriptors” in your “Model oKA Equation.” Example: 5-1 rule is used on 24 molecules you should have 4 in the “Descriptors per model” section. ** DON’T ROUND UP **</p></li><br />
<li><p>“No. of generations” can vary 200-500), but 200 is an okay default number to have.</p></li><br />
<li><p>“Models per Generation” should be at least 3 (better to have between 5-10).</p></li><br />
<li><p>Press “Run.”</p></li><br />
<li><p>Double Click on any of the cells in the first row.</p></li><br />
<li><p>A pop up window with a “Model aka equation” will pop up.</p></li><br />
<li><p>Copy and paste the model and information in the “()” onto Excel.</p></li><br />
<li><p>Close out of the window with the model information.</p></li><br />
<li><p>Copy the model and () information from all three rows.</p></li><br />
<li><p>These rows are the different models the BuildQSAR generated for you.</p></li></ol><br />
<br />
Descriptor Information:<br />
<br />
<ol start="56"><br />
<li><p>So each model (AKA equation) has a certain number of descriptors in it. In this example there are 3.</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>Y1 = - 1.6198 (± 1.1586) X269 + 0.0110 (± 0.0016) X631 - 0.0336 (± 0.0094) X634 + 0.8483 (± 0.3386)</p></li></ol><br />
</li><br />
<li><p>To find the meaning of these descriptors go to your Excel file and above your descriptor data create a row that is listed from X1 to X####. X1 is above the first descriptor, usually MW. The #’s are the number of descriptors in one row.</p></li><br />
<li><p>On Excel find the descriptors that are listed in the model. Find the abbreviation under the X value. Example X269 corresponded to the abbreviation “MATS3e”</p></li><br />
<li><p>Once you have all the descriptor abbreviations then go back to Dragon.</p></li><br />
<li><p>Select “ Descriptor Search”</p></li><br />
<li><p>Type an abbreviation you found from the Excel file.</p></li><br />
<li><p>Press “Search”</p></li><br />
<li><p>This will give the descriptor information.</p></li></ol><br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Test Set predictions<br />
<br />
Use obtained model to calculate “Predicted Experimental values” for the Test set.<br />
<br />
For that, find the descriptors selected by the model, find them among test set descriptors, and then calculate “Predicted Experimental values” by applying mathematical model from training set and values of selected descriptor for each compound in the test set.<br />
<br />
Do the same for all models, from 1 to 5 variables. Calculate predictive R2 values by correlation of “Experimental values” and Predicted Experimental values” in Excel (function CORR( )).<br />
<br />
Build a graph for all models from 1 to 5 variables, by drawing on axes X No of model and on axes Y the values R<sup>2</sup> for each model (separate lines for training and for the tests sets).<br />
<br />
Select the best model by finding the maximum value of R<sup>2</sup> for the test set models.<br />
<br />
Open Babel GUI:<br />
<br />
This section depends on conversion of one file type to another if needed. Usually this is done to get HOMO LUMO (Extra descriptors) from Hyperchem (another 3D software) which has file types of “*.hin” Also, this is used to convert file type to smiles notation if needed. In this example we are converting “*.mol” files to Smiles format notation and “*.hin” format. “*.hin” is for HyperChem<br /><br />
<br /><br />
<br /><br />
<br />
<br />
<ol start="64"><br />
<li><p>Open “Open Babel GUI”</p></li><br />
<li><p>On the top left of the window that opens chose the file type that you have. For example “mol—MDL MOL format is a common file extension.”</p></li><br />
<li><p>Once chosen you can select the way you want the data read in OpenBabel. In this example the “Use this format for all input files (ignore file extensions)” will be checked.</p></li><br />
<li><p>Clicking on the left side “…” box you can choose the file you would like converted.</p></li><br />
<li><p>Once the file was chosen it should be in the bar to the left of the “…” box</p></li><br />
<li><p>Under the very top right comment of “OUTPUT FORMAT” chose what you would like converted to. In this example chose “smiles - - SMILES format”</p></li><br />
<li><p>Check “Add Hydrogens (make explicit) in the middle column</p></li><br />
<li><p>Check “Generate 3D coordinates” in the middle column</p></li><br />
<li><p>Simply press convert and the data you want should be listed in the right column.</p></li><br />
<li><p>Copy the Smiles notation and save it onto another file such as excel or notepad. This is one example of what a smiles notation looks like “OC(F)(F)[C@H](F)C(F)(F)F”</p></li><br />
<li><p>The same step as 62 can be used to create the “*.hin” format notation. This is done by choosing “hin - - HyperChem HIN format” under OUTPUT FORMAT.</p></li><br />
<li><p>Make sure “Output below only (no output file)” and “Display in firefox” are NOT checked. These are on the right side column or AKA output column.</p></li><br />
<li><p>Before pressing Convert you need to specify an output file and location. A good idea is to name it exactly the same as the input fil except change the “.mol” ending file to “.hin” (THE “.hin” NEEDS TO BE INCLUDED IN THE OUTPUT FILE)</p></li><br />
<li><p>Press convert</p></li><br />
<li><p>The output file should be in the location with the name specified.</p></li></ol><br />
<br />
HyperChem:<br />
<br />
This section depends on the need for quantum descriptors such as HOMO, LUMO, and Total energy.<br />
<br />
<ol start="79"><br />
<li><p>Open “HyperChem”</p></li><br />
<li><p>Open a file that has “.hin” extension</p></li><br />
<li><p>Press “Setup” on the tool bar</p></li><br />
<li><p>Press “Semi-empirical”</p></li><br />
<li><p>Press “RM1”</p></li><br />
<li><p>Press “OK</p></li><br />
<li><p>Press “Compute” on the tool bar</p></li><br />
<li><p>Select “Geometry Optimization”</p></li><br />
<li><p>For regular work make sure “Polak –Ribiere (Conjugate gradiant)” is selected.</p></li><br />
<li><p>Have 0.1 in the top bar of RMS gradiant and in the bottom bar 600.</p></li><br />
<li><p>Have “In vacuo” selected</p></li><br />
<li><p>For the “Screen refresh period” use 1 for cycles.</p></li><br />
<li><p>Steps 78-81 should be default settings</p></li><br />
<li><p>Press “OK”</p></li><br />
<li><p>Once the optimization has stopped (Numbers on the bottom right or left have stopped moving) then select “Compute”</p></li><br />
<li><p>Select “Properties”</p></li><br />
<li><p>All the information in “Properties” can be used as quantum descriptors if needed</p></li><br />
<li><p>By Pressing “Details” you will get more information and these are also used as descriptors if needed.</p></li><br />
<li><p>Exit out of “Properties”</p></li><br />
<li><p>Press “Compute”</p></li><br />
<li><p>Select “Orbitals”</p></li><br />
<li><p>the screen that pops up shows 2-4 different columns, the left being Alpha orbitals and the right being Beta orbitals. The pink/purple column is simply the orbitals above the HOMO-LUMO Gap, and the green is simply below the HOMO LUMO Gap</p></li><br />
<li><p>Each line in these columns gives you the energy in “eV” for that orbital.</p></li><br />
<li><p>If you zoom into the bottom most line of the pink/purple column that is LUMO- Lowest unoccupied molecular orbital. If you select the bottom most line of the pink/purple column, it will give you a value in the “Energy” bar as “eV”. That is LUMO.</p></li><br />
<li><p>If you zoom into the top most line of the green column that is HOMO- Highest Occupied Molecular Orbital. If you select the top most like of the green column it will give you a value in the “Energy” bar as “eV”. That is HOMO</p></li><br />
<li><p>The difference of these energies is the HOMO-LUMO Gap.</p></li><br />
<li><p>If desired, you can simply “Plot” isosurfaces of the orbital that is selected by pressing the “Plot” button on the bottom left of the Orbitals panal.</p></li><br />
<li><p>Press “OK” when done with this screen.</p></li><br />
<li><p>To get further Descriptors</p></li><br />
<li><p>Select “Compute”</p></li><br />
<li><p>Select “ QSAR Properties” at the bottom</p></li><br />
<li><p>In this window you can select any of the properties that you desired to compute by simply selecting the property and press compute. The value will be shown at the bottom of the “QSAR Properties” window.</p></li><br />
<li><p>Scripts are also available to be written for Hyperchem if desired, you can do this via following Lesson 17 or 18 in this manual. <u>http://www.chemistry-software.com/pdf/Hyperchem_full_manual.pdf</u><br /><br />
<br /><br />
If manual doesn’t show up just google “Hyperchem Manual”</p></li><br />
<li><p>If It says your Molecule has issues before Geometry optimization such as Valence error go to Select</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>Select All</p></li><br />
<li><p>Select again</p></li><br />
<li><p>“Add H &amp; Model Build”</p></li></ol><br />
</li><br />
<li><p>This corrects the error</p></li><br />
<li><p>Additionally, If molecules are large and “Semi-Empirical” geometry optimization takes too much time then molecular mechanics is needed</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>When doing the molecular mechanics it is necessary to make sure the atom “type” is corrected.</p></li><br />
<li><p>To check if atom type is correct go to “Display”</p><br />
<ol style="list-style-type: lower-roman;"><br />
<li><p>“Labels”</p></li><br />
<li><p>“type”</p></li></ol><br />
</li><br />
<li><p>If you see “**” on the atoms then hyperchem needs to recalculate the atoms to get C, H, O, and other atoms.</p><br />
<ol style="list-style-type: lower-roman;"><br />
<li><p>Go to “setup”</p></li><br />
<li><p>“Molecular Mechanics”</p></li><br />
<li><p>Select “MM+”</p></li><br />
<li><p>Click on “Options”</p></li><br />
<li><p>Press OK</p></li><br />
<li><p>Click on “Components”</p></li><br />
<li><p>Press OK</p></li><br />
<li><p>Then press “OK” on the same window you selected “MM+”</p></li><br />
<li><p>You should get a popup window saying the atom types are going to be recalculated</p></li><br />
<li><p>Press “OK”</p></li></ol><br />
</li><br />
<li><p>This should change the “**” to letters such as C, H, O</p></li><br />
<li><p>If it doesn’t initially then try going to labels and pressing “none.” Then repeat steps 114 a-b</p></li><br />
<li><p>If that doesn’t work then select the semi-empirical. Once selected redo steps 114a-d</p></li><br />
<li><p>Use RHF not UHF</p></li></ol><br />
</li></ol><br />
<br />
<span id="_GoBack"></span><br /><br />
<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Dragon5_Procedure&diff=120Dragon5 Procedure2022-09-26T21:02:31Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== Dragon5 ======<br />
It is okay to use *.mol and *.hin, or any other format usable on Dragon, with no issues what so ever.<br />
<br />
# Open “Dragon5.exe,” exit out of the small windows that pup up.<br />
# Select “Calculate Descriptors” and then select all the “.mol2” files that you want to calculate. Use the ‘Ctrl’ button when selecting the files so all desired files can be selected.<br />
# Press the green check mark “OK” when all desired files are selected.<br />
# Choose the desired descriptors you want calculated for these molecule files. “X” means checked. Then press RUN.<br />
# Press Continue when the small window pops up.<br />
# A yellow window pops up and gives information about the calculations. If no errors are listed on the yellow window then exit out of it.<br />
# Select “Save Descriptors.”<br />
# Make sure “Constant Variables” &amp; “Near-Constant Variables” are selected “x.” Select “Pair Correlation” and pick .95 or something around that number.<br />
# Press “Save” and save as “.txt” file.<br />
# Leave Dragon open for future use.<br />
# Find the file you saved and open with Notepad++. You can do this by finding the file and right clicking on the file. Select “Edit with Notepad++” to open it in Notepad++.<br />
# In Notepad++, select all the information and copy onto an EXCEL worksheet. If desired you can delete the first 2 rows of the information on Notepad++. It is recommended to write the number of descriptors. This will help you later. It is listed in the 2<sup>nd</sup> row, 3<sup>rd</sup> column.<br />
# Once all the information is saved onto Excel then exit out of the Notepad++ file.</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=HyperChem&diff=119HyperChem2022-09-26T21:01:43Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== HyperChem ======<br />
This section depends on the need for quantum descriptors such as HOMO, LUMO, and Total energy.<br />
<br />
<ol><br />
<li><p>Open “HyperChem”</p></li><br />
<li><p>Open a file that has “.hin” extension</p></li><br />
<li><p>Press “Setup” on the tool bar</p></li><br />
<li><p>Press “Semi-empirical”</p></li><br />
<li><p>Press “RM1”</p></li><br />
<li><p>Press “OK</p></li><br />
<li><p>Press “Compute” on the tool bar</p></li><br />
<li><p>Select “Geometry Optimization”</p></li><br />
<li><p>For regular work make sure “Polak –Ribiere (Conjugate gradiant)” is selected.</p></li><br />
<li><p>Have 0.1 in the top bar of RMS gradiant and in the bottom bar 600.</p></li><br />
<li><p>Have “In vacuo” selected</p></li><br />
<li><p>For the “Screen refresh period” use 1 for cycles.</p></li><br />
<li><p>Steps 78-81 should be default settings</p></li><br />
<li><p>Press “OK”</p></li><br />
<li><p>Once the optimization has stopped (Numbers on the bottom right or left have stopped moving) then select “Compute”</p></li><br />
<li><p>Select “Properties”</p></li><br />
<li><p>All the information in “Properties” can be used as quantum descriptors if needed</p></li><br />
<li><p>By Pressing “Details” you will get more information and these are also used as descriptors if needed.</p></li><br />
<li><p>Exit out of “Properties”</p></li><br />
<li><p>Press “Compute”</p></li><br />
<li><p>Select “Orbitals”</p></li><br />
<li><p>the screen that pops up shows 2-4 different columns, the left being Alpha orbitals and the right being Beta orbitals. The pink/purple column is simply the orbitals above the HOMO-LUMO Gap, and the green is simply below the HOMO LUMO Gap</p></li><br />
<li><p>Each line in these columns gives you the energy in “eV” for that orbital.</p></li><br />
<li><p>If you zoom into the bottom most line of the pink/purple column that is LUMO- Lowest unoccupied molecular orbital. If you select the bottom most line of the pink/purple column, it will give you a value in the “Energy” bar as “eV”. That is LUMO.</p></li><br />
<li><p>If you zoom into the top most line of the green column that is HOMO- Highest Occupied Molecular Orbital. If you select the top most like of the green column it will give you a value in the “Energy” bar as “eV”. That is HOMO</p></li><br />
<li><p>The difference of these energies is the HOMO-LUMO Gap.</p></li><br />
<li><p>If desired, you can simply “Plot” isosurfaces of the orbital that is selected by pressing the “Plot” button on the bottom left of the Orbitals panal.</p></li><br />
<li><p>Press “OK” when done with this screen.</p></li><br />
<li><p>To get further Descriptors</p></li><br />
<li><p>Select “Compute”</p></li><br />
<li><p>Select “ QSAR Properties” at the bottom</p></li><br />
<li><p>In this window you can select any of the properties that you desired to compute by simply selecting the property and press compute. The value will be shown at the bottom of the “QSAR Properties” window.</p></li><br />
<li><p>Scripts are also available to be written for Hyperchem if desired, you can do this via following Lesson 17 or 18 in this manual. <u>http://www.chemistry-software.com/pdf/Hyperchem_full_manual.pdf</u></p><br />
<li><p>If manual doesn’t show up just google “Hyperchem Manual”</p></li></li><br />
<li><p>If It says your Molecule has issues before Geometry optimization such as Valence error go to Select</p><br />
<ul style="list-style-type: lower-alpha;"><br />
<li><p>Select All</p></li><br />
<li><p>Select again</p></li><br />
<li><p>“Add H &amp; Model Build”</p></li></ul><br />
</li><br />
<li><p>This corrects the error</p></li><br />
<li><p>Additionally, If molecules are large and “Semi-Empirical” geometry optimization takes too much time then molecular mechanics is needed</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>When doing the molecular mechanics it is necessary to make sure the atom “type” is corrected.</p></li><br />
<li><p>To check if atom type is correct go to “Display”</p><br />
<ol style="list-style-type: lower-roman;"><br />
<li><p>“Labels”</p></li><br />
<li><p>“type”</p></li></ol><br />
</li><br />
<li><p>If you see “**” on the atoms then hyperchem needs to recalculate the atoms to get C, H, O, and other atoms.</p><br />
<ol style="list-style-type: lower-roman;"><br />
<li><p>Go to “setup”</p></li><br />
<li><p>“Molecular Mechanics”</p></li><br />
<li><p>Select “MM+”</p></li><br />
<li><p>Click on “Options”</p></li><br />
<li><p>Press OK</p></li><br />
<li><p>Click on “Components”</p></li><br />
<li><p>Press OK</p></li><br />
<li><p>Then press “OK” on the same window you selected “MM+”</p></li><br />
<li><p>You should get a popup window saying the atom types are going to be recalculated</p></li><br />
<li><p>Press “OK”</p></li></ol><br />
</li><br />
<li><p>This should change the “**” to letters such as C, H, O</p></li><br />
<li><p>If it doesn’t initially then try going to labels and pressing “none.” Then repeat steps 114 a-b</p></li><br />
<li><p>If that doesn’t work then select the semi-empirical. Once selected redo steps 114a-d</p></li><br />
<li><p>Use RHF not UHF<br /></p></li></ol><br />
</li></ol></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=ChemSketchProcedure&diff=118ChemSketchProcedure2022-09-26T21:00:48Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== ChemSketch ======<br />
Can convert SMILES to structure and structure to SMILES.<br />
<br />
# Open Chem Sketch and click out of the little popup windows by clicking the X on the top right of the small window.<br />
# Draw the desired molecule in skeletal format.<br />
# Once the desired molecule is drawn then go to “Tools” and click “Clean Structure.” This will make it easier for the other programs to read what you created.<br />
# Next, go to “Tools” again and then “Generate.” You will have to generate the “Name of Structure,” “Smiles Notation,” and “InChi for Structure.”<br />
# Open a blank document on Excel. For the molecule created copy and paste the three things you generated. These three things will help keep your data organized. You may not need to refer to all three of these again but it is best to have them ready on hand as a precaution. You will use the “Name of Structure” for naming the files. ** Make sure you assign a number to each molecule. If you have 10 molecules just assign them numbers 1-10. Once again this will help with organizing your data. Organization is a big thing in this work!<br />
# After the three things you generated are on Excel and you assigned the molecule a number. Then delete the names off of chem sketch. Keep the structure and save it as an “MDL Molfiles (*.mol).” The file name can be whatever but it is preferred to follow this format. “Assigned # Name of Structure” Example… “33 2-ethenyl-5-nitrofuran.mol” The “.mol” should be automatically added once you select “MDL molfiles (*.mol).<br />
# Exit Chem Sketch.<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Jupyterlab_anaconda&diff=117Jupyterlab anaconda2022-09-26T21:00:09Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
{|<br />
|<br />
Log into Jupyter Lab/Notebook you need to open “''Anaconda prompt (anaconda3)”''<br />
|<br />
It is necessary to open anaconda prompt first by searching in your desktop windows search bar<br />
<br />
It looks like a regular terminal<br />
|-<br />
| <code>jupyter notebook</code><br />
| Type this into the anaconda prompt/terminal and you should<br />
|-<br />
| <code>jupyter notebook --notebook-dir D:/MachineLearningPythonBiodegradation</code><br />
| Because our directory fo jupyterlab isn’t where we want, I can specify which directory to open up and that is where my files can go. I am choosing this specific directory on my laptop since it is what I think is best.<br />
|-<br />
| <code>jupyter lab --notebook-dir D:/MachineLearningPythonBiodegradation</code><br />
| <br /><br />
|-<br />
| https://github.com/manuel-calzolari/sklearn-genetic<br />
| <br /><br />
|-<br />
| <code>conda install -c conda-forge sklearn-genetic</code><br />
| <br /><br />
|-<br />
| https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html<br />
| <br /><br />
|-<br />
| https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html<br />
| <br /><br />
|-<br />
|<br />
<code>bash Anaconda-latest-Linux-x86_64.sh</code><br />
<br />
<br /><br />
| <br /><br />
|-<br />
| <code>eval &quot;$(/gpfs1/home/meade.erickson/anaconda3/bin/conda shell.bash hook)&quot;</code><br />
| Used this to get conda to work in my environment<br />
|-<br />
|<br />
Using putty: <code>prime.ccast.ndsu.edu</code><br />
| <br /><br />
|-<br />
| <code>ls -lisanh</code><br />
| gives information on the files in whole directory<br />
|-<br />
| <code>ls -lisanh filename</code><br />
| gives information on the specific file<br />
|-<br />
| <code>ls -lisan</code><br />
| gives information on the files in whole directory<br />
|-<br />
| <code>ls -lisan filename</code><br />
| gives information on the specific file<br />
|-<br />
| <code>babel –ixyz file.xyz –opdb file.pdb xyz to pdb</code><br />
|<br />
Using babel, converts the input file of .xyz to outputfile of .pdb<br />
|}<br />
<br />
<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=ChemDraw_Procedure&diff=116ChemDraw Procedure2022-09-26T20:56:16Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== ChemDraw ======<br />
can convert name of structure to a structure.<br />
<br />
# ChemDraw has a special feature that if given a name of a polymer, it will generate the monomer unit for you.<br />
# Open ChemDraw Professional<br />
# Go to “Structure”<br />
# Select “Convert Name to Structure”<br />
# Type in the name of the polymer without “poly” for example. “Polybutane” only type “butane.”<br />
# It will give you a generalized version. This will assist in giving you the correct monomer unit or just generalized version.<br />
# *** ONLY HAVE STURCTURE IN “*.MOL” FILE*** Delete the text after the model is generated because the text will cause issues later<br />
# NEXT STEPS ARE FOR CREATION OF STRUCTURE AND NAME REGULAR WAY LIKE DRAWING<br />
# Draw the desired molecule<br />
# Select the model by pressing the logo near the top left of the screen [[File:ChemDraw_Procedure_HTML_c603a81bb8df177f.png|22x25px]]<br />
# Once selected go to “Structure” and click “Clean up Structure.” This will make it easier for the other programs to read what you created.<br />
# To generate the name of this structure go to “Structure” and press “Convert structure to name.”<br />
# Open a blank document on Excel. For the molecule created copy and paste the name you generated. This will help keep your data organized.<br />
# After the model is generated and the name is on Excel, assign a number to the molecule on excel. Then delete the names off of ChemDraw. Keep the structure and save it as an “MDL Molfiles (*.mol).” The file name can be whatever but it is preferred to follow this format. “Assigned # Name of Structure” Example… “33 2-ethenyl-5-nitrofuran.mol” The “.mol” should be automatically added once you select “MDL molfiles (*.mol).<br />
# *** ONLY HAVE STURCTURE IN “*.MOL” FILE*** Delete the text after the model is generated because the text will cause issues later<br />
# Exit Chem Draw.</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=Avogadro_Procedure&diff=115Avogadro Procedure2022-09-26T20:55:38Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== Avogadro ======<br />
# After doing these steps for all the molecules you wanted to create then open the program Avogadro.<br />
# Go to “File” and “Open” to bring a molecule you created in ChemSketch to Avogadro. Find your molecule that you want to bring to Avogadro and select it to open it. Click “Yes” when a small window pops up about 3D coordinates and a rough sketch.<br />
# Once the molecule is on Avogadro then go to “Extensions” and select “Optimize geometry.”<br />
# * There is another way to Optimize Geometry How do we do it again???<br />
# Save the file as “Sybyl Mol2 (*.mol2)”<br />
# Exit Avogadro.<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=BuildQsar_Procedure&diff=114BuildQsar Procedure2022-09-26T20:55:12Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== BuildQSAR ======<br />
# Open “BuildQSAR.”<br />
# Go to “File” then “New.”<br />
# Add a Title to “Dataset Title.”<br />
# Change the number of “Compounds” to the amount in your data. Change the number of “Descriptors” you have. The number of descriptors was obtained from the Notepad++ information.<br />
# Click “Ok.”<br />
# Input observed data into the yellow column. The yellow column is for the observed information from other sources such as research papers.<br />
# ** The observed data should be put into Excel at this point.<br />
# Copy and paste the descriptor information from Excel into the blue cells of BuildQSAR.<br />
# Go to “QSAR” then “Variable Selection” then “Systematic Search” or “Genetic Algorithm.” (note: Choose Genetic Algorithm only when you need 4, 5 or higher number of variables in the model).<br />
# A small popup window will pop up. Make sure the 2 boxes under “Cross Validation” are checked. The correlation criteria can change but if uncertain on a number then put 0.6 as default.<br />
# “Descriptors per Model,” this is usually calculated using the 5-1 rule. The 5-1 rule relates the number of molecules you have to the number “Variables AKA Descriptors” in your “Model oKA Equation.” Example: 5-1 rule is used on 24 molecules you should have 4 in the “Descriptors per model” section. ** DON’T ROUND UP **<br />
# “No. of generations” can vary 200-500), but 200 is an okay default number to have.<br />
# “Models per Generation” should be at least 3 (better to have between 5-10).<br />
# Press “Run.”<br />
# Double Click on any of the cells in the first row.<br />
# A pop up window with a “Model aka equation” will pop up.<br />
# Copy and paste the model and information in the “()” onto Excel.<br />
# Close out of the window with the model information.<br />
# Copy the model and () information from all three rows.<br />
# These rows are the different models the BuildQSAR generated for you.<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=AutoDock&diff=113AutoDock2022-09-26T20:54:32Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
by Kristen Patnode<br />
<br />
=== AutoDockTools Procedure ===<br />
Updated 03/09/2020<br />
<br />
====== '''Protein Preparation''' ======<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Once you’ve opened AutoDockTools, go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down and select “Read Molecule”</span></span></span></span></span><br />
#* Then open your receptor/protein pdb file:<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_224e67a633bba335.png|561x274px]]<br />
<br />
<ol start="2"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to the </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">select </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">click on </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">set selection level</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">atom</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_7950b6eb0106e110.png|572x323px]]</p></li><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none">'''''<span style="background: transparent">OPTIONAL STEP:</span>'''''</span></span> <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Edit</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Delete</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Delete Selected Atoms</span>'''</span></span></span></p><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">In this way you can remove any unnecessary or unwanted atoms from the file.</span></span></span></span></span></p></li></li></ol><br />
<br />
<ol start="4"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Add hydrogens to the structure if needed</span></span></span></span></span></p><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Edit</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Hydrogens </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Add</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> then in the pop up window, choose </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Polar Only</span>'''</span></span></span></p></li></li></ol><br />
<br />
<ol start="5"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Grid</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">macro molecule</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Choose</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select your protein file in the pop up window.</span></span></span></span></span></p></li></ol><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_771dad82e05bc903.png|251x297px]] [[File:AutodDock_klpatnode_docking_instructions_030920_HTML_4f8e01f7de2be83.png|431x234px]]<br />
<br />
<ol start="6"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Save the protein as a pdbqt file</span></span></span></span></span></p><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">save</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> Write PDBQT</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span> [[File:AutodDock_klpatnode_docking_instructions_030920_HTML_e5fb90772da674ca.png|624x275px]]</p></li></li></ol><br />
<br />
Choose a file name and location and then click '''OK'''<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_81cc39e642d996e5.png|490x367px]]<br />
<br />
<ol start="7"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Create your grid box for where the ligand can dock</span></span></span></span></span></p><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Grid</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down, </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Grid Box, </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">move box and adjust size as necessary.</span></span></span></span></span></p></li></li></ol><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_3ca196597d18df76.png|624x404px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="background: transparent">RECORD THE VALUES OF YOUR GRID BOX (size and center)</span></span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="background: transparent"> You will need to use these values in the command prompt portion of the process!!!</span></span></span><br />
<br />
====== <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Ligand Preparation</span>'''</span></span></span> ======<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to the </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Ligand</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Input</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">open</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose your ligand pdb file</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span> [[File:AutodDock_klpatnode_docking_instructions_030920_HTML_352bd88d1295bdc7.png|428x336px]]<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Again under the </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Ligand </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Torsion Tree </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Choose Torsion</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> click on the bonds you want to make rotatable </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">done</span>'''</span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_4eee221b4b18f4f3.png|624x405px]]<br />
<br />
<ol start="3"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Save ligand as a pdbqt file like you did for protein</span></span></span></span></span></p><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Ligand-> </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Output</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">-></span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">save as PDBQT</span>'''</span></span></span></p></li></li></ol><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_2a5defc5d7d31176.png|425x323px]]<br />
<br />
====== '''Docking''' ======<br />
'''1'''. Rename the Scripps Research folder to: The_Scripps_Research_Institute<br />
<br />
*The underscores help since command prompt prefers not to have spaces<br />
<br />
'''2'''. on the first command line type the following with your respective/appropriate folder names and usernames.<br />
<br />
<code>cd \Users\john.doe\documents\foldername\The_Scripps_Research_Institute</code><br />
<br />
'''''This is easiest if you copy and paste the route directly from file explorer (see image)'''''<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_81e0c3f958a9e516.png|624x230px]]<br />
<br />
'''3.''' open the vina folder by typing the following with your respective/appropriate folder names and usernames.<br />
<br />
<code>cd \Users\john.doe\documents\foldername\The-Scripps_Research_Institute\Vina</code><br />
<br />
'''''This is again easiest if you copy and paste the route directly from file explorer (see image)'''''<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_1d3b8cbfe19a290e.png|624x207px]]<br />
<br />
'''4.''' Create a file in notepad in the following order<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_eae2b7da623781da.png|329x256px]]<br />
<br />
Where “####” are the numbers you get from your grid box dimensions completed above in the protein preparation.<br />
<br />
'''5.''' At this point your command prompt should have the vina folder open and look like the image below<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_b7378e520e2da49d.png|624x323px]]<br />
<br />
The next step is to open your notepad file. To do so, type the following in the command prompt:<br />
<br />
<code>vina.exe --config notepadfilename.txt --log log.txt</code><br />
<br />
Use whatever filename you saved the notepad file as. For my files I name them “conf_date” and put the date of the updated version to make it easier to keep track.<br />
<br />
'''''*preceding “config” and the first “log” are two hyphens! *'''''<br />
<br />
At the end of the script it should say “writing output … done.[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_9721fd503846c5a9.png|757x492px]]<br />
<br />
'''6.''' See the output file in your Vina folder as filename_out.pdbqt<br />
<br />
<u>'''Analysis of the Output'''</u><br />
<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Read Molecule</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose the output file that was generated from command prompt above. --_out.pdbqt </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">open</span>'''</span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_7072a865d8a291e8.png|432x171px]]<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_3360dd7a2abe320c.png|207x174px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Again go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Read Molecule </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose your protein.pdbqt file </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">open</span>'''</span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_cb41f38d950f8697.png|624x288px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">display </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Interactions</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Set Molecule List</span>'''</span></span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">The first should be your ligand and second will be your protein file. </span></span></span></span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none">''<span style="font-weight: normal"><span style="background: transparent">See image below for reference</span></span>''</span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_a0c441fe84dab521.png|624x329px]]<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_7b1b055b735d9cd0.png|624x399px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="background: transparent"> </span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">You can then choose background color and adjust the colors of the protein and ligand as desired.</span></span></span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_4fe58df3a986ea39.png|624x464px]]<br />
<br />
<ol start="4"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">To save the image go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Save</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Save Image As </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> name the output file </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">OK</span>'''</span></span></span></p></li></ol><br />
<br />
====== <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">PyMol Visualization</span>'''</span></span></span> ======<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Open the protein structure (PDB format) first</span></span></span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_3f87dda5a8014c27.png|593x316px]]<br />
<br />
<ol start="2"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Open the AutoDock output file from command prompt calculations.</span></span></span></span></span></p></li></ol><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_485d60e792c19228.png|624x325px]]<br />
<br />
<ol start="3"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">All</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> preset</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">ligand sites </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> cartoon</span>'''</span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_b3a221e8140627fd.png|624x326px]]</p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">You should see something like this:</span></span></span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_d6baaed4a39c8b70.png|624x321px]]<br />
<br />
<ol start="4"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="background: transparent"> <span style="font-style: normal"><span style="font-weight: normal">To measure distances between select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Wizard </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> measurement </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">then select the atom you want to start measuring from and then the atom you are measuring to.</span></span></span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_fa1f80a37b0ae3e7.png|624x355px]]</p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">*when you are done measuring, hit </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">done </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">at the bottom right otherwise everything you click will create a measurement.*</span></span></span></span></span><br />
<br />
<ol start="5"><br />
<li><p><span id="_gjdgxs"></span> <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">For additional visualization techniques, feel free to ask me (Kristen ) any questions. Otherwise, YouTube has a plethora of videos on this topic that are very helpful!</span></span></span></span></span></p></li></ol></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=AutoDock&diff=112AutoDock2022-09-26T20:48:00Z<p>Sysadmin: Created page with "__TOC__ Kristen Patnode AutoDockTools Procedure Updated 03/09/2020 <u>'''Protein Preparation'''</u> # <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Once you’ve opened AutoDockTools, go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span st..."</p>
<hr />
<div>__TOC__<br />
<br />
Kristen Patnode<br />
<br />
AutoDockTools Procedure<br />
<br />
Updated 03/09/2020<br />
<br />
<u>'''Protein Preparation'''</u><br />
<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Once you’ve opened AutoDockTools, go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down and select “Read Molecule”</span></span></span></span></span><br />
<br />
Then open your receptor/protein pdb file<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_224e67a633bba335.png|561x274px]]<br />
<br />
<ol start="2"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to the </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">select </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">click on </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">set selection level</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">atom</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span> [[File:AutodDock_klpatnode_docking_instructions_030920_HTML_7950b6eb0106e110.png|572x323px]]</p></li><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none">'''''<span style="background: transparent">OPTIONAL STEP</span>'''''</span></span></p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Edit</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Delete</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Delete Selected Atoms</span>'''</span></span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">In this way you can remove any unnecessary or unwanted atoms from the file.</span></span></span></span></span><br />
<br />
<ol start="4"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Add hydrogens to the structure if needed</span></span></span></span></span></p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Edit</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Hydrogens </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Add</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> then in the pop up window, choose </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Polar Only</span>'''</span></span></span><br />
<br />
<ol start="5"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Grid</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">macro molecule</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Choose</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select your protein file in the pop up window.</span></span></span></span></span></p></li></ol><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_771dad82e05bc903.png|251x297px]] [[File:AutodDock_klpatnode_docking_instructions_030920_HTML_4f8e01f7de2be83.png|431x234px]]<br />
<br />
<ol start="6"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Save the protein as a pdbqt file</span></span></span></span></span></p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">save</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> Write PDBQT</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span> [[File:AutodDock_klpatnode_docking_instructions_030920_HTML_e5fb90772da674ca.png|624x275px]]<br />
<br />
Choose a file name and location and then click '''OK'''<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_81cc39e642d996e5.png|490x367px]]<br />
<br />
<ol start="7"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Create your grid box for where the ligand can dock</span></span></span></span></span></p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Grid</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Grid Box </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> move box and adjust size as necessary.</span></span></span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_3ca196597d18df76.png|624x404px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none">'''''<span style="background: transparent">RECORD THE VALUES OF YOUR GRID BOX (size and center)</span>'''''</span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none">'''''<span style="background: transparent"> You will need to use these values in the command prompt portion of the process!!!</span>'''''</span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><u>'''<span style="background: transparent">Ligand Preparation</span>'''</u></span></span></span><br />
<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to the </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Ligand</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Input</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">open</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose your ligand pdb file</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span> [[File:AutodDock_klpatnode_docking_instructions_030920_HTML_352bd88d1295bdc7.png|428x336px]]<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Again under the </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Ligand </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">drop down </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Torsion Tree </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Choose Torsion</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> click on the bonds you want to make rotatable </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">done</span>'''</span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_4eee221b4b18f4f3.png|624x405px]]<br />
<br />
<ol start="3"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Save ligand as a pdbqt file like you did for protein</span></span></span></span></span></p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Ligand </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Output</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">save as PDBQT</span>'''</span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_2a5defc5d7d31176.png|425x323px]]<br />
<br />
<u>'''Docking'''</u><br />
<br />
'''1'''. Rename the Scripps Research folder to:<br />
<br />
The_Scripps_Research_Institute<br />
<br />
*The underscores help since command prompt prefers not to have spaces<br />
<br />
'''2'''. on the first command line type the following with your respective/appropriate folder names and usernames.<br />
<br />
cd \Users\john.doe\documents\foldername\The_Scripps_Research_Institute<br />
<br />
'''''This is easiest if you copy and paste the route directly from file explorer (see image)'''''<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_81e0c3f958a9e516.png|624x230px]]<br />
<br />
'''3.''' open the vina folder by typing the following with your respective/appropriate folder names and usernames.<br />
<br />
cd \Users\john.doe\documents\foldername\The-Scripps_Research_Institute\Vina<br />
<br />
'''''This is again easiest if you copy and paste the route directly from file explorer (see image)'''''<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_1d3b8cbfe19a290e.png|624x207px]]<br />
<br />
'''4.''' Create a file in notepad in the following order<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_eae2b7da623781da.png|329x256px]]<br />
<br />
Where “####” are the numbers you get from your grid box dimensions completed above in the protein preparation.<br />
<br />
'''5.''' At this point your command prompt should have the vina folder open and look like the image below<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_b7378e520e2da49d.png|624x323px]]<br />
<br />
The next step is to open your notepad file. To do so, type the following in the command prompt:<br />
<br />
vina.exe --config notepadfilename.txt --log log.txt<br />
<br />
Use whatever filename you saved the notepad file as. For my files I name them “conf_date” and put the date of the updated version to make it easier to keep track.<br />
<br />
'''''*preceding “config” and the first “log” are two hyphens! *'''''<br />
<br />
At the end of the script it should say “writing output … done.[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_9721fd503846c5a9.png|757x492px]]<br />
<br />
'''6.''' See the output file in your Vina folder as filename_out.pdbqt<br />
<br />
<u>'''Analysis of the Output'''</u><br />
<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Read Molecule</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose the output file that was generated from command prompt above. --_out.pdbqt </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">open</span>'''</span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_7072a865d8a291e8.png|432x171px]]<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_3360dd7a2abe320c.png|207x174px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Again go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Read Molecule </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> choose your protein.pdbqt file </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">open</span>'''</span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_cb41f38d950f8697.png|624x288px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">display </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Interactions</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Set Molecule List</span>'''</span></span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">The first should be your ligand and second will be your protein file. </span></span></span></span></span><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none">''<span style="font-weight: normal"><span style="background: transparent">See image below for reference</span></span>''</span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_a0c441fe84dab521.png|624x329px]]<br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_7b1b055b735d9cd0.png|624x399px]]<br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="background: transparent"> </span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">You can then choose background color and adjust the colors of the protein and ligand as desired.</span></span></span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_4fe58df3a986ea39.png|624x464px]]<br />
<br />
<ol start="4"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">To save the image go to </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">file</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Save</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Save Image As </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent"> name the output file </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">🡪</span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">OK</span>'''</span></span></span></p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><u>'''<span style="background: transparent">PyMol Visualization</span>'''</u></span></span></span><br />
<br />
# <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Open the protein structure (PDB format) first</span></span></span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_3f87dda5a8014c27.png|593x316px]]<br />
<br />
<ol start="2"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Open the AutoDock output file from command prompt calculations.</span></span></span></span></span></p></li></ol><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_485d60e792c19228.png|624x325px]]<br />
<br />
<ol start="3"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">Select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">All</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> preset</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">ligand sites </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> cartoon</span>'''</span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_b3a221e8140627fd.png|624x326px]]</p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">You should see something like this:</span></span></span></span></span><br />
<br />
[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_d6baaed4a39c8b70.png|624x321px]]<br />
<br />
<ol start="4"><br />
<li><p><span style="font-variant: normal"><span style="text-decoration: none"><span style="background: transparent"> <span style="font-style: normal"><span style="font-weight: normal">To measure distances between select </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">Wizard </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">🡪</span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent"> measurement </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">then select the atom you want to start measuring from and then the atom you are measuring to.</span></span></span></span></span>[[File:AutodDock_klpatnode_docking_instructions_030920_HTML_fa1f80a37b0ae3e7.png|624x355px]]</p></li></ol><br />
<br />
<span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">*when you are done measuring, hit </span></span></span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal">'''<span style="background: transparent">done </span>'''</span></span></span><span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">at the bottom right otherwise everything you click will create a measurement.*</span></span></span></span></span><br />
<br />
<ol start="5"><br />
<li><p><span id="_gjdgxs"></span> <span style="font-variant: normal"><span style="text-decoration: none"><span style="font-style: normal"><span style="font-weight: normal"><span style="background: transparent">For additional visualization techniques, feel free to ask me (Kristen ) any questions. Otherwise, YouTube has a plethora of videos on this topic that are very helpful!</span></span></span></span></span></p></li></ol></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=ADF_ReaxFF&diff=111ADF ReaxFF2022-09-26T20:46:48Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
How to ADF<br />
<br />
<code>dos2unix ''filename''</code><br />
<br />
<code>chmod 750 ''filename''</code><br />
<br />
<code>thunder.ccast.ndsu.edu</code><br />
<br />
====== Generating a system in ADF ======<br />
<ol><br />
<li><p>Locate the ADF application on the desktop.</p></li><br />
<li><p>If it says “ADFjobs” follow steps 3-5, if it says “ADF” skip to step 6</p></li><br />
<li><p>Click on the icon “ADFjobs” on the desktop to open ADF</p></li><br />
<li><p>Click on “SCM” in the upper left corner</p></li><br />
<li><p>Under “SCM” click “New Input”</p></li><br />
<li><p>Click on the yellow “ADF” button.</p></li><br />
<li><p>From the drop-down menu, click “ReaxFF”</p></li><br />
<li><p>Next to “Force field:” click the folder icon to open a force field file.</p></li><br />
<li><p>For example, a commonly used force field is “CHON2017_weak.ff”</p></li><br />
<li><p>Change the number of iterations, time steps, temperature and others depending on your reaction system.</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>For density convergence simulations it is common to use 100,000+ iterations.</p><br />
<ol style="list-style-type: lower-roman;"><br />
<li><p>For density convergence simulations, next to “Method:” click on the drop-down menu and select “NPT Berendsen.”</p></li><br />
<li><p>Also, set the pressure to 0.101 MPa.</p></li></ol><br />
</li><br />
<li><p>For other simulations, 40,000 iterations are usually enough.</p></li></ol><br />
</li><br />
<li><p>Everything can be changed by clicking on the tabs such as “ADF”, “Main,” “Model,” “Properties,” “Details,” “MultiLevel”</p><br />
<li><p>You just must know what your changing and why by reading what others have done via papers</p></li></li><br />
<li><p>For adding molecules, click “Edit” in the top left corner.</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>Select Builder</p></li></ol><br />
</li><br />
<li><p>In Builder you can change the dimensions of the volume you are using for your experiment by changing the values under the table, “Lattice vectors.”</p></li><br />
<li><p>Under “Molecules” you can add copies of different molecules.</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>Set the number of molecules you would like to add by changing to number next to “Fill box with:”</p></li><br />
<li><p>You can select which molecules to add by typing the name into the bar next to “copies of:” or you can click on the folder icon to open an XYZ file of a molecule saved on the computer.</p></li></ol><br />
</li><br />
<li><p>While setting the dimensions, check on the density of the system on the bottom left of the builder menu. The density should remain &lt;1.5 g/mL. If the density is too high, ADF will not fill the box with all the molecules. Occasionally ADF has difficulties filling the box with all the molecules if the molecules are particularly large.</p></li><br />
<li><p>Review and save your file.</p></li></ol><br />
<br />
====== Submitting a job using CCAST ======<br />
<br />
# To run your job with CCAST you will need to use WinSCP and PuTTY or SSH Secure File Transfer. For using WinSCP and PuTTY, follow steps 18 to 38. For using SSH Secure File transfer, skip to step 39.<br />
# Before you can submit your job using PuTTY you must move your files to WinSCP. To start, open WinSCP from the desktop or by searching for the application. Once the window has opened, enter “thunder.ccast.ndsu.edu” into the “Host Name” bar.<br />
# Enter your username and password for CCAST into the applicable blanks.<br />
# Now, we must import the files into the directory we will be using to run the job. We will be running jobs from /gpfs1/scratch/firstname.lastname<br />
# Navigate to your folder in WinSCP and copy your files into the folder that has your name. It may be wise to create various folders in your name to organize your files. Once you have pasted your files they are in correct area to be used by PuTTY.<br />
# For using PuTTY, open the application from the desktop or by searching the application in the search bar.<br />
# When you open PuTTY enter “thunder.ccast.ndsu.edu” into the “Host Name” bar.<br />
# Once the command window has opened, enter your login username and password for CCAST.<br />
# To connect to the Scratch directory to find your files type “cd $SCRATCH” the space is important.<br />
# You can type the command “ls” to see what files are in your current directory.<br />
# Use the command “cd filename” to open the targeted file. Tip: instead of manually typing the file name you can double click on the name to highlight it and then hit the shift and insert keys at the same time to paste the file name into the command line.<br />
# Continue using the commands “ls” and “cd filename” to navigate to the folder containing your files of interest. If you wish to go back one directory use the command “cd ..”<br />
# Once you have located your files you must enter several commands to convert your files into a format that CCAST can understand. The first command is “dos2unix filename”. For running ReaxFF calculations the completed command will be “dos2unix Epoxyamine.run”.<br />
# The next command to type is “chmod 750 filename”. In our case it will be “chmod 750 Epoxyamine.run”<br />
# To check if the file has been converted correctly type “ls”. If the file is converted the name should be in green.<br />
# If the file is not green, check your syntax. If the file is green, submit the job by typing “qsub filename”. In our case, type “qsub Reaxff_exl.pbs”. Remember the shift+insert shortcut to save time.<br />
# To check if the file is running, enter the command “qstat -u firstname.lastname”. A table should popup that will tell you many things, among which are the status of the job and the run time. Under the “S” column there can be several different letters. “R” means running and “E” means error for example. Under the “Elap Time” column you can see the time the job has been running.<br />
# To check on what is exactly occurring while the job is running, in WinSCP if you navigate to the same directory the file is being ran from. Then, click the refresh button (the icon is two green arrows pointed in a circle). This will refresh the directory and show the files being created.<br />
# The file “summary.txt” is a useful file to see how the job is running. Open the file to see the current status and values of your file.<br />
# The file “molfra.out” allows you to see the current species present in your simulation.<br />
# To check if the job is finished you can open the file “summary.txt” and look at the left most column, “Iteration”. When you scroll down to the bottom of the page and that number matches the number of iterations you specified when you are creating the file the job is done.<br />
# Copy the file “reaxout.kf” to a folder on the computer to start the data analysis.<br />
<br />
====== Older stuff from Meade ======<br />
<br />
# Open SSH Secure File transfer<br />
## Log on and open a File Transfer Window<br />
# In ADF Save file in newfolder because they only get generated this will get ADF and pid and run files<br />
# Transfer from desktop to ccast. Use by dragging into file ccast<br />
# Go to terminal got to folder<br />
# dos2unix filename.run<br />
# chmod 750 filename.run<br />
# qsub filename.pbs<br />
# the reaxout.kf file is the thing we want<br />
# in adf go to scm and kf browser<br />
# file open the kf<br />
# go to scm movies and then properties<br />
# qstat -u meade.erickson<br />
# ls -sh<br />
# for troubleshooting use dos2unix on the pbs file and run files again. l sometimes it works.<br />
<br />
====== Data analysis in ADF ======<br />
# Once you have downloaded saved the reaxout.kf file you can start using ADF for analysis.<br />
# Open the icon “ADFjobs” and click on “SCM” in the upper left area.<br />
# Under “SCM” click on “KFBrowser”.<br />
# Once the new window has opened, click “File” then “Open…”.<br />
# Search through your files until you find the folder with your reaxout.kf file in it. The file may not be visible at first. In the bottom right corner of the window click on the drop down bar that says “Any known file type (*.t21;rkf;” Then, select “All files (*.*)” Your reaxout.kf file should now appear if it wasn’t already. Open the file.<br />
# Once the file has loaded in to the KFBrowser, click on “SCM” then click “Movie”.<br />
# Once this window pops up you have many things you can now investigate. Under the “Properties” tab you can find many aspects of the simulation to investigate.<br />
# One of the most important properties of the system is the “Molecule Fractions”. Click on this property to see any additional molecules that have formed.<br />
# To see a graph of a certain molecules over time, click on the left most box under the graph column. To save the data from a graph that is displayed click on “Graph” then “Save As XY”<br />
# Play around with other properties to figure out other aspects of the system as needed.<br />
<br />
====== Using CCAST and Python Scripts to Submit Bond Boosting Jobs ======<br />
# To start, you must download the package from the ADF website. Go to this link <u>https://www.scm.com/doc/Tutorials/Advanced/ReaxFF_polymers_bond_boost.html</u><br />
#* About 2/3 down the page look for where it says “<span style="background: #ffffff">Download the workflow package from </span><u>[https://downloads.scm.com/distr/EPOXY-POLYMERIZATION.zip <span style="background: #ffffff">here</span>]</u>” Download the package as a .zip file. (If you right-click and select open hyperlink on “<u>[https://downloads.scm.com/distr/EPOXY-POLYMERIZATION.zip <span style="background: #ffffff">here</span>]</u>” in this article it might actually download it for you).<br />
<span id="_Hlk43277817"></span><ol start="2"><br />
<li><p>Once it has downloaded you must extract the packages. Right click on the downloaded file, “EPOXY-POLYMERIZATION.zip”, and click “Extract all…” Select a destination and save it.</p></li><br />
<li><p>Now, move the unzipped folder to your WinSCP scratch folder.</p></li><br />
<li><p>Now we are ready for an example for how to run the program.</p></li><br />
<li><p>Navigate through your scratch folder using the commands “cd” and “ls” until you get to get to the “resins-hardeners” folder inside “EPOXY-POLYMERIZATION”</p></li><br />
<li><p>Copy and paste the file “pack-box.py” from the scripts folder into the “resins-hardeners” folder.</p></li><br />
<li><p>Once you are inside that folder type the command “python pack-box.py resin hardener ratio dimension”</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>For your individual command replace “resin” and “hardener” with your compounds you are using.</p></li><br />
<li><p>Replace “ratio” with a number for your ratio between the resin and hardener.</p></li><br />
<li><p>Replace “dimension” with a number for how large you want the box to be. The box is a cube with sides 15 Angstroms*dimension long.</p></li></ol><br />
</li><br />
<li><p>An example for that command is “python pack-box.py BisF.bgf DETDA.bgf 2 1”</p></li><br />
<li><p>After the command has been executed, 2 new files should be formed in the “resins-hardeners” folder.</p></li><br />
<li><p>Copy and paste those files into the “orig” directory inside “EPOXY-POLYMERIZATION”.</p></li><br />
<li><p>Once pasted, remove the suffix “.bgf” from one of the files.</p></li><br />
<li><p>Copy and paste to file “bond-boost.py” from the “scripts” folder into the “EPOXY-POLYMERIZATION” folder.</p></li><br />
<li><p>Locate the file “python_job.pbs” and copy and paste the file into the “EPOXY-POLYMERIZATION” folder as well.</p></li><br />
<li><p>Now, open the “bond-boost.py” file and change the file names under “initial_geometries”</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>Change the name of the files after “orig/” to match the name of the files created.</p></li></ol><br />
</li><br />
<li><p>Now, connect to the main “EPOXY-POLYMERIZATION” folder and type the command “qsub python_jobs.pbs”</p></li><br />
<li><p>This command will create a working directory where new files will be generated.</p></li><br />
<li><p>To check the status, open the folder “plams_workdirXXX” and open the log files.</p></li><br />
<li><p>When it is done, the last line will say “Goodbye”</p></li><br />
<li><p>The program will create many folders and files in the working directory. Many .kf will be created and each one can be analyzed to show a different step in the process.</p></li></ol></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=OpenBabelGUI&diff=110OpenBabelGUI2022-09-26T20:43:45Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
This section depends on conversion of one file type to another if needed. Usually this is done to get HOMO LUMO (Extra descriptors) from Hyperchem (another 3D software) which has file types of “*.hin” Also, this is used to convert file type to smiles notation if needed. In this example we are converting “*.mol” files to Smiles format notation and “*.hin” format. “*.hin” is for HyperChem.<br /><br />
# Open “Open Babel GUI”<br />
# On the top left of the window that opens chose the file type that you have. For example “mol—MDL MOL format is a common file extension.”<br />
# Once chosen you can select the way you want the data read in OpenBabel. In this example the “Use this format for all input files (ignore file extensions)” will be checked.<br />
# Clicking on the left side “…” box you can choose the file you would like converted.<br />
# Once the file was chosen it should be in the bar to the left of the “…” box<br />
# Under the very top right comment of “OUTPUT FORMAT” chose what you would like converted to. In this example chose “smiles - - SMILES format”<br />
# Check “Add Hydrogens (make explicit) in the middle column<br />
# Check “Generate 3D coordinates” in the middle column<br />
# Simply press convert and the data you want should be listed in the right column.<br />
# Copy the Smiles notation and save it onto another file such as excel or notepad. This is one example of what a smiles notation looks like “OC(F)(F)[C@H](F)C(F)(F)F”<br />
# The same step as 62 can be used to create the “*.hin” format notation. This is done by choosing “hin - - HyperChem HIN format” under OUTPUT FORMAT.<br />
# Make sure “Output below only (no output file)” and “Display in firefox” are NOT checked. These are on the right side column or AKA output column.<br />
# Before pressing Convert you need to specify an output file and location. A good idea is to name it exactly the same as the input fil except change the “.mol” ending file to “.hin” (THE “.hin” NEEDS TO BE INCLUDED IN THE OUTPUT FILE)<br />
# Press convert<br />
# The output file should be in the location with the name specified.</div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=QSARINS_Procedure&diff=109QSARINS Procedure2022-09-26T20:42:59Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
8/13/20 by Marvellous Ngongang<br />
<br />
# Locate the QSARINS (QI) application on the desktop.<br />
# Click on the QSARINS screen to proceed.<br />
# To import data, select the icon “file” and click on import data.<br />
# Select the file you want to import.<br />
# If necessary, you can select the “exclude if constant” option or the “exclude if correlational.” If neither are necessary, then proceed to click on the icon “import.”<br />
# Click on the icon “Analysis.” Then, click on “Data Setup.”<br />
# Data setup<br />
## Under variable setup i. Click on the first cell under the status column then select the icon “Select response.”<br />
## Click on the second cell under status column. While holding down the shift button, then scroll down to the last cell and click on it while holding down.<br />
## If necessary, select the “normalize var."<br />
# Under Splitting setup, click on the “Training comp.” drop down and select number 4.<br />
# Click on the “Prediction comp.” drop down and select number 1<br />
## Select “ok.”<br />
# Click on the icon “Analysis.” Then, click on “Variable selection and models calculation.”<br />
# Select the “GA until” box and proceed with the standard numbers imputed. Then, select “Calculate.”<li><p>Click on the icon “View and select model” located to the left of “PBT Index.”</p></li><br />
<li><p>“View and select model”</p><br />
# <li><p>Right click on the maroon colored cell and click on “Select all models”</p></li><li><p>Right click on the screen then click on “External model validation”</p></li><li><p>Right click on the screen then click on “Internal model validation”</p></li><li><p>Select “LMO,” “Y-scramble,” “Rand. Desc.,” and “Rand. Resp.”</p></li></li><li><p>Proceed with the standard numbers imputed. Then, select “Validate.”</p></li><li><p>Click on the icon “View selected model” below the “Sorted by” drop down.</p></li><li><p>Click on the icon “View experimental endpoint vs. Predicted by model equation” located to the right of the save button.</p></li><li><p>Click on the icon “ View Williams plot-Predictions by model equation.”</p><br />
# <li><p>Change the Std. Dev. to 3.</p></li></li><li><p>Click on the icon “Y scramble.”</p></li><li><p>Select the “X min,” “X max,” “Y min,” and “Y, max” box. Change the “Y min” number to 0.</p></li></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=WEKA_Steps_for_Loading_Data&diff=108WEKA Steps for Loading Data2022-09-26T20:39:18Z<p>Sysadmin: Revised formatting</p>
<hr />
<div>__TOC__<br />
<br />
====== Steps for Loading Data into WEKA ======<br />
ARFF format consists of three parts: '''@RELATION''', '''@ATTRIBUTE''' and '''@DATA'''.<br />
<br />
* @RELATION name<br />
* @ATTRIBUTE descriptor_name data_type (numeric, nominal …)<br />
* @DATA: numbers (integer or real) or strings<span id="_GoBack"></span>General rules for ARFF file can be found here: https://www.cs.waikato.ac.nz/ml/weka/arff.html<br />
<br />
====== In Excel: ======<br />
* File 1: the format in this file is three columns for: “'''@ATTRIBUTE'''”, the descriptors’ names, and “'''NUMERIC'''”<br />
** Open the txt data file in Excel. Make sure you are searching from “All Files”.<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_4ec5c9075dd9cb51.png|512x288px]] [[File:WEKA_Steps_for_Loading_Data_HTML_ae35e05dbfaa6f7c.gif|31x38px|Shape1]]<br />
<br />
''Figure 1, Opening your data file''<br />
<br />
* When '''Text Import Wizard''' prompts, choose '''Delimited and''' click on '''Next''' ('''step 1'''), check the '''Tab''' box ('''step 2''') and click on '''Finish'''. (Leave the rest as default unless it is necessary to change.)<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_d288ed8722be4fdc.png|393x300px]] [[File:WEKA_Steps_for_Loading_Data_HTML_369e56d90bf80141.gif|32x31px|Shape3]] [[File:WEKA_Steps_for_Loading_Data_HTML_bef7f76831abec8b.gif|14x32px|Shape2]]<br />
<br />
''Figure 2, Step 1''<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_cd41ce313e7118b1.png|379x288px]] [[File:WEKA_Steps_for_Loading_Data_HTML_859149fac76acd64.gif|30x19px|Shape5]] [[File:WEKA_Steps_for_Loading_Data_HTML_6092b35113c6658c.gif|19x33px|Shape4]]<br />
<br />
''Figure 3, Step 2''<br />
<br />
* Create three blank columns.<br />
* Copy the descriptors’ names (they are at the first row of your data file, normally) and paste them in the vertical form by using '''“Transpose”''' pasting option '''to the second column'''.<br />
* Make sure the '''cell format''' is '''Text''' before the next step.<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_5e70e797a5c2cba0.png|313x156px]] [[File:WEKA_Steps_for_Loading_Data_HTML_634c3e41e8a46d53.gif|59x34px|Shape6]]<br />
<br />
''Figure 4, Cell format''<br />
<br />
* For the first and third columns: make equal number of rows as the second column has for “@ATTRIBUTE” and “NUMERIC”, respectively.<br />
* Example:<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_e8fa2497f6fc4487.png|198x337px]]<br />
<br />
''Figure 5, the three columns''<br />
* File 2: Here we separate the @DATA part (only numbers) of data.<br />
** Keep only the numbers needed and delete everything else.<br />
** Save it in '''CSV (comma delimited)''' format. ('''Click YES''' when asks “Some features in your workbook … Do you want to keep using that format?”)<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_d4bc2328d2c3083f.png|279x64px]] [[File:WEKA_Steps_for_Loading_Data_HTML_4fc01181394c3a93.gif|43x28px|Shape7]]<br />
<br />
''Figure 6, Save as CSV''<br />
<br />
====== In Notepad++: ======<br />
* File A:<br />
** In the first row, create two columns: @RELATION and a title for the relation name (which are just separated by a space).<br />
** ''(For aesthetics:'' leave the second row blank.)<br />
** Copy the three columns in File 1 of Excel and paste into row3.<br />
** Example:<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_1f0af9e857ead597.png|247x204px]]<br />
<br />
''Figure 7, start of the File A''<br />
<br />
* File A (cont):<br />
** Depending on your data, the last @ATTRIBUTE row will need to be the response variable AKA the thing you are trying to predict. (CATEGORICAL may require this, NUMERIC may not<br />
** After all ATTRIBUTE information has been pasted leave 1 row blank ''(For aesthetics'')<br />
** After blank row Type “@DATA”.<br />
** ''(For aesthetics:'' For the next row after “@DATA”, leave it blank.)<br />
** Example:<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_ca4125bd73bd6170.png|460x172px]]<br />
<br />
''Figure 8, middle of File A''<br />
* File B:<br />
** Open File 2 from Notepad++ (and you should see the data are separated by commas).<br />
** Example:<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_964b046e1656a7f.png|555x225px]]<br />
<br />
''Figure <span style="background: #c0c0c0">9</span>, File 2 open with NotePad++''<br />
<br />
* Copy and paste all the data to File A.<br />
<br />
* Already returned to File A:<br />
** Save the file as “arff” format (by adding “.arff” at the end of the file name).<br />
<br />
'''The file is ready to open and run in WEKA (yay).'''<br />
<br />
<br />
PS: (1) “%” would give you error if included in the descriptors’ names, specifically such as '''“%N” or “%O”''', after “@ATTRIBUTE”. The error would occur because whatever after “%” is considered as comment, then in WEKA, it would be interpreted as a missing information for the descriptor name and data type. Simply remove “%” would avoid errors. (2)Also make sure the number of descriptors matches with the number of data in the “@DATA” section. (If you have 10 lines of @ATTRIBUTE + descriptors’ names, there should have 10 numbers in each line in @DATA part in the Notepad++.)<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5&diff=107WorkShop Avogadro BuildQSAR ChemSketch Dragon52022-09-26T20:32:59Z<p>Sysadmin: /* Descriptor Information: */</p>
<hr />
<div>__TOC__<br />
<br />
'''NDSU-KU Symposium QSAR Workshop'''<br />
<br />
'''Purpose''': Using ChemSketch, Avogadro, Dragon5, and Build QSAR programs to generate structures, descriptors, and models.<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_19a26003ffd9c4c4.gif|578x431px]]<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_ce016f9083bf0bf2.png|308x329px]]<br />
<br />
====== '''1.''' '''ChemSketch''': (Molecule Generation) ======<br />
# Open ChemSketch and click out of the little popup windows by clicking the X on the top right of the small window.<br />
# Next, go to “Tools” and then “Generate.” Select “Structure from SMILES.”<br />
# Type in the SMILES Notation from above.<br />
# Push “OK.”<br />
# Left click where you want to make the molecule.<br />
# Once the desired molecule is drawn then go to “Tools” and click “Clean Structure.” This will make it easier for the other programs to read what you created.<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save ChemSketch structure as “#.mol” where # is the number associated with the table above.<br />
# Repeat steps to create the desired number of structures. Save them as individual *.mol Files.<br />
# Exit ChemSketch.<br />
<br />
====== '''2. Avogadro''': (Molecule Optimization) ======<br />
# Open the program Avogadro.<br />
# Go to “File” and “Open” to bring a molecule you created in ChemSketch to Avogadro. Find your molecule that you want to bring to Avogadro and select it to open it.<br />
# A small window will pop up about 3D coordinates and a rough sketch. Click “Yes.”<br />
# Once the molecule is on Avogadro then go to “Extensions” and select “Optimize geometry.”<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save the files as *.mol2 for all the individual molecules you created.<br />
# Exit Avogadro.<br />
<br />
====== '''3. Dragon5''': (Descriptor Generation &amp; Data Setup) ======<br />
# Open “Dragon5.exe,” exit out of the small windows that pup up.<br />
# Select “Calculate Descriptors” and then select all the “.mol2” files that you want to calculate. Use the ‘Ctrl’ button when selecting the files so all desired files can be selected.<br />
# Press the green check mark “OK” when all desired files are selected.<br />
# Choose the desired descriptors you want calculated for these molecule files. “X” means checked. Then press RUN.<br />
# Press Continue when the small window pops up.<br />
# A yellow window pops up and gives information about the calculations. If no errors are listed on the yellow window then exit out of it.<br />
# Select “Save Descriptors.”<br />
# Make sure “Constant Variables” &amp; “Near-Constant Variables” are selected “x.” Select “Pair Correlation” and pick .95 or something around that number.<br />
# Press “Save” and save as “DESCRIPTORS.txt” file.<br />
# Exit Dragon<br />
# Find the “DESCRIPTOR.txt” file you saved and open with Notepad++. You can do this by finding the file and right clicking on the file. Select “Edit with Notepad++” to open it in Notepad++.<br />
# ***VERY IMPORTANT Write the number of descriptors, the number is listed in 2<sup>nd</sup> Row, 3<sup>rd</sup> Column.<br />
# Leave the DESCRIPTOR.txt file open<br />
<br />
====== '''4. BuildQSAR''': (Model Development) ======<br />
<ol><br />
<li><p>Open “BuildQSAR.”</p></li><br />
<li><p>Go to “File” then “New.”</p></li><br />
<li><p>Add a Title to “Dataset Title.”</p></li><br />
<li><p>Change the number of “Compounds” to the number of molecules you have created AND ADD 4. For example, if you have 10 molecules put in the number 14. Change the number of “Descriptors” to the number you got from the *.txt file AND ADD 2. So if you have 632 descriptors then type 634.</p></li><br />
<li><p>Click “Ok.”</p></li><br />
<li><p>From your DESCRIPTOR.txt file select all the information by pressing “Ctrl + A.” Copy the selected information by pressing “Ctrl + C”</p></li><br />
<li><p>In buildQSAR select the top left BLUE square and paste the data by “right clicking” using the mouse and select “PASTE”</p></li><br />
<li><p>Delete the columns and rows that have words and incomplete information. Usually that is the first 4 rows. You can do this by left clicking One data point from those columns or rows then select “Dataset”, “Remove.” The Row is removed if you press “compound.” The Column is removed if you press “descriptor”.</p></li><br />
<li><p>In the yellow (Y1) column input the data from the Table above which shows the Log(10)Toxicity</p></li><br />
<li><p>Go to “QSAR” then “Variable Selection” then “Systematic Search” or “Genetic Algorithm.” (note: Choose Genetic Algorithm only when you need 4, 5 or higher number of variables in the model).</p></li><br />
<li><p>A small popup window will pop up. Make sure the 2 boxes under “Cross Validation” are checked.</p></li><br />
<li><p>The correlation criteria can change but if uncertain on a number then put 0.6 as default.</p></li><br />
<li><p>For “Genetic Algorithm:”</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>“Descriptors per Model,” this is usually calculated using the 5-1 rule. The 5-1 rule relates the number of molecules you have to the number “Variables AKA Descriptors” in your “Model oKA Equation.” Example: 5-1 rule is used on 24 molecules you should have 4 in the “Descriptors per model” section. ** DON’T ROUND UP **</p></li><br />
<li><p><span id="_GoBack"></span> “No. of generations” can vary 200-500), but 200 is an okay to use as a default number.</p></li><br />
<li><p>“Models per Generation” should be at least 3 (better to have between 5-10).</p></li></ol><br />
</li><br />
<li><p>Press “Run.”</p></li><br />
<li><p>When complete “Double Click” on any of the cells in the first row.</p></li><br />
<li><p>That is your developed models.</p></li></ol><br />
<br />
=== '''Descriptor Information''': ===<br />
<br />
# So each model (AKA equation) has a certain number of descriptors in it. In this example there are 3.<br />
<br />
<math display="block">Y = -1.6198 (\pm 1.1586) X_{269} + 0.0110 (\pm 0.0016) X_{631} - 0.0336 (\pm 0.0094) X_{634} + 0.8483 (\pm 0.3386)</math><br />
<br />
<li>To find the meaning of these descriptors go to your Excel file and above your descriptor data create a row that is listed from X1 to X####. X1 is above the first descriptor, usually MW. The #’s are the number of descriptors in one row.<br />
</li><br />
<li>On Excel find the descriptors that are listed in the model. Find the abbreviation under the X value. Example X269 corresponded to the abbreviation “MATS3e”<br />
</li><br />
<li>Once you have all the descriptor abbreviations then go back to Dragon.<br />
</li><br />
<li>Select “ Descriptor Search”<br />
</li><br />
<li>Type an abbreviation you found from the Excel file.<br />
</li><br />
<li>Press “Search”<br />
</li><br />
<li>This will give the descriptor information.<br />
</li><br />
<li>You can also click on this link to find what the abbreviation means<br />
# <li><u>http://www.talete.mi.it/products/dragon_molecular_descriptor_list.pdf</u><br />
</li></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5&diff=106WorkShop Avogadro BuildQSAR ChemSketch Dragon52022-09-26T20:32:39Z<p>Sysadmin: /* Descriptor Information: math formula fix */</p>
<hr />
<div>__TOC__<br />
<br />
'''NDSU-KU Symposium QSAR Workshop'''<br />
<br />
'''Purpose''': Using ChemSketch, Avogadro, Dragon5, and Build QSAR programs to generate structures, descriptors, and models.<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_19a26003ffd9c4c4.gif|578x431px]]<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_ce016f9083bf0bf2.png|308x329px]]<br />
<br />
====== '''1.''' '''ChemSketch''': (Molecule Generation) ======<br />
# Open ChemSketch and click out of the little popup windows by clicking the X on the top right of the small window.<br />
# Next, go to “Tools” and then “Generate.” Select “Structure from SMILES.”<br />
# Type in the SMILES Notation from above.<br />
# Push “OK.”<br />
# Left click where you want to make the molecule.<br />
# Once the desired molecule is drawn then go to “Tools” and click “Clean Structure.” This will make it easier for the other programs to read what you created.<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save ChemSketch structure as “#.mol” where # is the number associated with the table above.<br />
# Repeat steps to create the desired number of structures. Save them as individual *.mol Files.<br />
# Exit ChemSketch.<br />
<br />
====== '''2. Avogadro''': (Molecule Optimization) ======<br />
# Open the program Avogadro.<br />
# Go to “File” and “Open” to bring a molecule you created in ChemSketch to Avogadro. Find your molecule that you want to bring to Avogadro and select it to open it.<br />
# A small window will pop up about 3D coordinates and a rough sketch. Click “Yes.”<br />
# Once the molecule is on Avogadro then go to “Extensions” and select “Optimize geometry.”<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save the files as *.mol2 for all the individual molecules you created.<br />
# Exit Avogadro.<br />
<br />
====== '''3. Dragon5''': (Descriptor Generation &amp; Data Setup) ======<br />
# Open “Dragon5.exe,” exit out of the small windows that pup up.<br />
# Select “Calculate Descriptors” and then select all the “.mol2” files that you want to calculate. Use the ‘Ctrl’ button when selecting the files so all desired files can be selected.<br />
# Press the green check mark “OK” when all desired files are selected.<br />
# Choose the desired descriptors you want calculated for these molecule files. “X” means checked. Then press RUN.<br />
# Press Continue when the small window pops up.<br />
# A yellow window pops up and gives information about the calculations. If no errors are listed on the yellow window then exit out of it.<br />
# Select “Save Descriptors.”<br />
# Make sure “Constant Variables” &amp; “Near-Constant Variables” are selected “x.” Select “Pair Correlation” and pick .95 or something around that number.<br />
# Press “Save” and save as “DESCRIPTORS.txt” file.<br />
# Exit Dragon<br />
# Find the “DESCRIPTOR.txt” file you saved and open with Notepad++. You can do this by finding the file and right clicking on the file. Select “Edit with Notepad++” to open it in Notepad++.<br />
# ***VERY IMPORTANT Write the number of descriptors, the number is listed in 2<sup>nd</sup> Row, 3<sup>rd</sup> Column.<br />
# Leave the DESCRIPTOR.txt file open<br />
<br />
====== '''4. BuildQSAR''': (Model Development) ======<br />
<ol><br />
<li><p>Open “BuildQSAR.”</p></li><br />
<li><p>Go to “File” then “New.”</p></li><br />
<li><p>Add a Title to “Dataset Title.”</p></li><br />
<li><p>Change the number of “Compounds” to the number of molecules you have created AND ADD 4. For example, if you have 10 molecules put in the number 14. Change the number of “Descriptors” to the number you got from the *.txt file AND ADD 2. So if you have 632 descriptors then type 634.</p></li><br />
<li><p>Click “Ok.”</p></li><br />
<li><p>From your DESCRIPTOR.txt file select all the information by pressing “Ctrl + A.” Copy the selected information by pressing “Ctrl + C”</p></li><br />
<li><p>In buildQSAR select the top left BLUE square and paste the data by “right clicking” using the mouse and select “PASTE”</p></li><br />
<li><p>Delete the columns and rows that have words and incomplete information. Usually that is the first 4 rows. You can do this by left clicking One data point from those columns or rows then select “Dataset”, “Remove.” The Row is removed if you press “compound.” The Column is removed if you press “descriptor”.</p></li><br />
<li><p>In the yellow (Y1) column input the data from the Table above which shows the Log(10)Toxicity</p></li><br />
<li><p>Go to “QSAR” then “Variable Selection” then “Systematic Search” or “Genetic Algorithm.” (note: Choose Genetic Algorithm only when you need 4, 5 or higher number of variables in the model).</p></li><br />
<li><p>A small popup window will pop up. Make sure the 2 boxes under “Cross Validation” are checked.</p></li><br />
<li><p>The correlation criteria can change but if uncertain on a number then put 0.6 as default.</p></li><br />
<li><p>For “Genetic Algorithm:”</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>“Descriptors per Model,” this is usually calculated using the 5-1 rule. The 5-1 rule relates the number of molecules you have to the number “Variables AKA Descriptors” in your “Model oKA Equation.” Example: 5-1 rule is used on 24 molecules you should have 4 in the “Descriptors per model” section. ** DON’T ROUND UP **</p></li><br />
<li><p><span id="_GoBack"></span> “No. of generations” can vary 200-500), but 200 is an okay to use as a default number.</p></li><br />
<li><p>“Models per Generation” should be at least 3 (better to have between 5-10).</p></li></ol><br />
</li><br />
<li><p>Press “Run.”</p></li><br />
<li><p>When complete “Double Click” on any of the cells in the first row.</p></li><br />
<li><p>That is your developed models.</p></li></ol><br />
<br />
=== '''Descriptor Information''': ===<br />
<br />
# So each model (AKA equation) has a certain number of descriptors in it. In this example there are 3.<br />
<br />
<math display="block">$$Y = -1.6198 (\pm 1.1586) X_{269} + 0.0110 (\pm 0.0016) X_{631} - 0.0336 (\pm 0.0094) X_{634} + 0.8483 (\pm 0.3386)$$</math><br />
<br />
<li>To find the meaning of these descriptors go to your Excel file and above your descriptor data create a row that is listed from X1 to X####. X1 is above the first descriptor, usually MW. The #’s are the number of descriptors in one row.<br />
</li><br />
<li>On Excel find the descriptors that are listed in the model. Find the abbreviation under the X value. Example X269 corresponded to the abbreviation “MATS3e”<br />
</li><br />
<li>Once you have all the descriptor abbreviations then go back to Dragon.<br />
</li><br />
<li>Select “ Descriptor Search”<br />
</li><br />
<li>Type an abbreviation you found from the Excel file.<br />
</li><br />
<li>Press “Search”<br />
</li><br />
<li>This will give the descriptor information.<br />
</li><br />
<li>You can also click on this link to find what the abbreviation means<br />
# <li><u>http://www.talete.mi.it/products/dragon_molecular_descriptor_list.pdf</u><br />
</li></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5&diff=105WorkShop Avogadro BuildQSAR ChemSketch Dragon52022-09-26T19:54:12Z<p>Sysadmin: revising page for consistent formatting</p>
<hr />
<div>__TOC__<br />
<br />
'''NDSU-KU Symposium QSAR Workshop'''<br />
<br />
'''Purpose''': Using ChemSketch, Avogadro, Dragon5, and Build QSAR programs to generate structures, descriptors, and models.<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_19a26003ffd9c4c4.gif|578x431px]]<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_ce016f9083bf0bf2.png|308x329px]]<br />
<br />
====== '''1.''' '''ChemSketch''': (Molecule Generation) ======<br />
# Open ChemSketch and click out of the little popup windows by clicking the X on the top right of the small window.<br />
# Next, go to “Tools” and then “Generate.” Select “Structure from SMILES.”<br />
# Type in the SMILES Notation from above.<br />
# Push “OK.”<br />
# Left click where you want to make the molecule.<br />
# Once the desired molecule is drawn then go to “Tools” and click “Clean Structure.” This will make it easier for the other programs to read what you created.<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save ChemSketch structure as “#.mol” where # is the number associated with the table above.<br />
# Repeat steps to create the desired number of structures. Save them as individual *.mol Files.<br />
# Exit ChemSketch.<br />
<br />
====== '''2. Avogadro''': (Molecule Optimization) ======<br />
# Open the program Avogadro.<br />
# Go to “File” and “Open” to bring a molecule you created in ChemSketch to Avogadro. Find your molecule that you want to bring to Avogadro and select it to open it.<br />
# A small window will pop up about 3D coordinates and a rough sketch. Click “Yes.”<br />
# Once the molecule is on Avogadro then go to “Extensions” and select “Optimize geometry.”<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save the files as *.mol2 for all the individual molecules you created.<br />
# Exit Avogadro.<br />
<br />
====== '''3. Dragon5''': (Descriptor Generation &amp; Data Setup) ======<br />
# Open “Dragon5.exe,” exit out of the small windows that pup up.<br />
# Select “Calculate Descriptors” and then select all the “.mol2” files that you want to calculate. Use the ‘Ctrl’ button when selecting the files so all desired files can be selected.<br />
# Press the green check mark “OK” when all desired files are selected.<br />
# Choose the desired descriptors you want calculated for these molecule files. “X” means checked. Then press RUN.<br />
# Press Continue when the small window pops up.<br />
# A yellow window pops up and gives information about the calculations. If no errors are listed on the yellow window then exit out of it.<br />
# Select “Save Descriptors.”<br />
# Make sure “Constant Variables” &amp; “Near-Constant Variables” are selected “x.” Select “Pair Correlation” and pick .95 or something around that number.<br />
# Press “Save” and save as “DESCRIPTORS.txt” file.<br />
# Exit Dragon<br />
# Find the “DESCRIPTOR.txt” file you saved and open with Notepad++. You can do this by finding the file and right clicking on the file. Select “Edit with Notepad++” to open it in Notepad++.<br />
# ***VERY IMPORTANT Write the number of descriptors, the number is listed in 2<sup>nd</sup> Row, 3<sup>rd</sup> Column.<br />
# Leave the DESCRIPTOR.txt file open<br />
<br />
====== '''4. BuildQSAR''': (Model Development) ======<br />
<ol><br />
<li><p>Open “BuildQSAR.”</p></li><br />
<li><p>Go to “File” then “New.”</p></li><br />
<li><p>Add a Title to “Dataset Title.”</p></li><br />
<li><p>Change the number of “Compounds” to the number of molecules you have created AND ADD 4. For example, if you have 10 molecules put in the number 14. Change the number of “Descriptors” to the number you got from the *.txt file AND ADD 2. So if you have 632 descriptors then type 634.</p></li><br />
<li><p>Click “Ok.”</p></li><br />
<li><p>From your DESCRIPTOR.txt file select all the information by pressing “Ctrl + A.” Copy the selected information by pressing “Ctrl + C”</p></li><br />
<li><p>In buildQSAR select the top left BLUE square and paste the data by “right clicking” using the mouse and select “PASTE”</p></li><br />
<li><p>Delete the columns and rows that have words and incomplete information. Usually that is the first 4 rows. You can do this by left clicking One data point from those columns or rows then select “Dataset”, “Remove.” The Row is removed if you press “compound.” The Column is removed if you press “descriptor”.</p></li><br />
<li><p>In the yellow (Y1) column input the data from the Table above which shows the Log(10)Toxicity</p></li><br />
<li><p>Go to “QSAR” then “Variable Selection” then “Systematic Search” or “Genetic Algorithm.” (note: Choose Genetic Algorithm only when you need 4, 5 or higher number of variables in the model).</p></li><br />
<li><p>A small popup window will pop up. Make sure the 2 boxes under “Cross Validation” are checked.</p></li><br />
<li><p>The correlation criteria can change but if uncertain on a number then put 0.6 as default.</p></li><br />
<li><p>For “Genetic Algorithm:”</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>“Descriptors per Model,” this is usually calculated using the 5-1 rule. The 5-1 rule relates the number of molecules you have to the number “Variables AKA Descriptors” in your “Model oKA Equation.” Example: 5-1 rule is used on 24 molecules you should have 4 in the “Descriptors per model” section. ** DON’T ROUND UP **</p></li><br />
<li><p><span id="_GoBack"></span> “No. of generations” can vary 200-500), but 200 is an okay to use as a default number.</p></li><br />
<li><p>“Models per Generation” should be at least 3 (better to have between 5-10).</p></li></ol><br />
</li><br />
<li><p>Press “Run.”</p></li><br />
<li><p>When complete “Double Click” on any of the cells in the first row.</p></li><br />
<li><p>That is your developed models.</p></li></ol><br />
<br />
=== '''Descriptor Information''': ===<br />
<br />
# So each model (AKA equation) has a certain number of descriptors in it. In this example there are 3.<br />
<br />
<math display="block">Y = -1.6198 (\pm 1.1586) X_{269} + 0.0110 (\pm 0.0016) X_{631} - 0.0336 (\pm 0.0094) X_{634} + 0.8483 (\pm 0.3386})</math><br />
<br />
<li>To find the meaning of these descriptors go to your Excel file and above your descriptor data create a row that is listed from X1 to X####. X1 is above the first descriptor, usually MW. The #’s are the number of descriptors in one row.<br />
</li><br />
<li>On Excel find the descriptors that are listed in the model. Find the abbreviation under the X value. Example X269 corresponded to the abbreviation “MATS3e”<br />
</li><br />
<li>Once you have all the descriptor abbreviations then go back to Dragon.<br />
</li><br />
<li>Select “ Descriptor Search”<br />
</li><br />
<li>Type an abbreviation you found from the Excel file.<br />
</li><br />
<li>Press “Search”<br />
</li><br />
<li>This will give the descriptor information.<br />
</li><br />
<li>You can also click on this link to find what the abbreviation means<br />
# <li><u>http://www.talete.mi.it/products/dragon_molecular_descriptor_list.pdf</u><br />
</li></li></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5&diff=104WorkShop Avogadro BuildQSAR ChemSketch Dragon52022-09-26T19:21:34Z<p>Sysadmin: Imported from text file</p>
<hr />
<div>__TOC__<br />
<br />
<div title="header"><br />
<br />
<span style="background: #c0c0c0">4</span><br />
<br />
<br /><br />
<br />
<br />
<br />
</div><br />
'''NDSU-KU Symposium QSAR Workshop'''<br />
<br />
'''Purpose''': Using ChemSketch, Avogadro, Dragon5, and Build QSAR programs to generate structures, descriptors, and models.<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_19a26003ffd9c4c4.gif|578x431px]]<br />
<br />
[[File:WorkShop_Avogadro_BuildQSAR_ChemSketch_Dragon5__HTML_ce016f9083bf0bf2.png|308x329px]]<br />
<br />
'''ALL programs should be on the DESKTOP or you can search for it in the Windows Search Bar'''<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
# '''ChemSketch''': (Molecule Generation)<br />
<br />
# Open ChemSketch and click out of the little popup windows by clicking the X on the top right of the small window.<br />
# Next, go to “Tools” and then “Generate.” Select “Structure from SMILES.”<br />
# Type in the SMILES Notation from above.<br />
# Push “OK.”<br />
# Left click where you want to make the molecule.<br />
# Once the desired molecule is drawn then go to “Tools” and click “Clean Structure.” This will make it easier for the other programs to read what you created.<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save ChemSketch structure as “#.mol” where # is the number associated with the table above.<br />
# Repeat steps to create the desired number of structures. Save them as individual *.mol Files.<br />
# Exit ChemSketch.<br />
<br />
'''2. Avogadro''': (Molecule Optimization)<br />
<br />
# Open the program Avogadro.<br />
# Go to “File” and “Open” to bring a molecule you created in ChemSketch to Avogadro. Find your molecule that you want to bring to Avogadro and select it to open it.<br />
# A small window will pop up about 3D coordinates and a rough sketch. Click “Yes.”<br />
# Once the molecule is on Avogadro then go to “Extensions” and select “Optimize geometry.”<br />
# Then go to “File” then “Save as” and choose the “KU WorkShop Practice” Folder on the DESKTOP.<br />
# Save the files as *.mol2 for all the individual molecules you created.<br />
# Exit Avogadro.<br />
<br />
'''3. Dragon5''': (Descriptor Generation &amp; Data Setup)<br />
<br />
# Open “Dragon5.exe,” exit out of the small windows that pup up.<br />
# Select “Calculate Descriptors” and then select all the “.mol2” files that you want to calculate. Use the ‘Ctrl’ button when selecting the files so all desired files can be selected.<br />
# Press the green check mark “OK” when all desired files are selected.<br />
# Choose the desired descriptors you want calculated for these molecule files. “X” means checked. Then press RUN.<br />
# Press Continue when the small window pops up.<br />
# A yellow window pops up and gives information about the calculations. If no errors are listed on the yellow window then exit out of it.<br />
# Select “Save Descriptors.”<br />
# Make sure “Constant Variables” &amp; “Near-Constant Variables” are selected “x.” Select “Pair Correlation” and pick .95 or something around that number.<br />
# Press “Save” and save as “DESCRIPTORS.txt” file.<br />
# Exit Dragon<br />
# Find the “DESCRIPTOR.txt” file you saved and open with Notepad++. You can do this by finding the file and right clicking on the file. Select “Edit with Notepad++” to open it in Notepad++.<br />
# ***VERY IMPORTANT Write the number of descriptors, the number is listed in 2<sup>nd</sup> Row, 3<sup>rd</sup> Column.<br />
# Leave the DESCRIPTOR.txt file open<br />
<br />
'''4. BuildQSAR''': (Model Development)<br />
<br />
<ol><br />
<li><p>Open “BuildQSAR.”</p></li><br />
<li><p>Go to “File” then “New.”</p></li><br />
<li><p>Add a Title to “Dataset Title.”</p></li><br />
<li><p>Change the number of “Compounds” to the number of molecules you have created AND ADD 4. For example, if you have 10 molecules put in the number 14. Change the number of “Descriptors” to the number you got from the *.txt file AND ADD 2. So if you have 632 descriptors then type 634.</p></li><br />
<li><p>Click “Ok.”</p></li><br />
<li><p>From your DESCRIPTOR.txt file select all the information by pressing “Ctrl + A.” Copy the selected information by pressing “Ctrl + C”</p></li><br />
<li><p>In buildQSAR select the top left BLUE square and paste the data by “right clicking” using the mouse and select “PASTE”</p></li><br />
<li><p>Delete the columns and rows that have words and incomplete information. Usually that is the first 4 rows. You can do this by left clicking One data point from those columns or rows then select “Dataset”, “Remove.” The Row is removed if you press “compound.” The Column is removed if you press “descriptor”.</p></li><br />
<li><p>In the yellow (Y1) column input the data from the Table above which shows the Log(10)Toxicity</p></li><br />
<li><p>Go to “QSAR” then “Variable Selection” then “Systematic Search” or “Genetic Algorithm.” (note: Choose Genetic Algorithm only when you need 4, 5 or higher number of variables in the model).</p></li><br />
<li><p>A small popup window will pop up. Make sure the 2 boxes under “Cross Validation” are checked.</p></li><br />
<li><p>The correlation criteria can change but if uncertain on a number then put 0.6 as default.</p></li><br />
<li><p>For “Genetic Algorithm:”</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>“Descriptors per Model,” this is usually calculated using the 5-1 rule. The 5-1 rule relates the number of molecules you have to the number “Variables AKA Descriptors” in your “Model oKA Equation.” Example: 5-1 rule is used on 24 molecules you should have 4 in the “Descriptors per model” section. ** DON’T ROUND UP **</p></li><br />
<li><p><span id="_GoBack"></span> “No. of generations” can vary 200-500), but 200 is an okay to use as a default number.</p></li><br />
<li><p>“Models per Generation” should be at least 3 (better to have between 5-10).</p></li></ol><br />
</li><br />
<li><p>Press “Run.”</p></li><br />
<li><p>When complete “Double Click” on any of the cells in the first row.</p></li><br />
<li><p>That is your developed models.</p></li></ol><br />
<br />
<br /><br />
<br /><br />
<br />
<br />
'''WorkShop Ends Here'''<br />
<br />
'''The information below is for you to look at if you want.'''<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
'''Descriptor Information''':<br />
<br />
<ol start="17"><br />
<li><p>So each model (AKA equation) has a certain number of descriptors in it. In this example there are 3.</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p>Y1 = - 1.6198 (± 1.1586) X269 + 0.0110 (± 0.0016) X631 - 0.0336 (± 0.0094) X634 + 0.8483 (± 0.3386)</p></li></ol><br />
</li><br />
<li><p>To find the meaning of these descriptors go to your Excel file and above your descriptor data create a row that is listed from X1 to X####. X1 is above the first descriptor, usually MW. The #’s are the number of descriptors in one row.</p></li><br />
<li><p>On Excel find the descriptors that are listed in the model. Find the abbreviation under the X value. Example X269 corresponded to the abbreviation “MATS3e”</p></li><br />
<li><p>Once you have all the descriptor abbreviations then go back to Dragon.</p></li><br />
<li><p>Select “ Descriptor Search”</p></li><br />
<li><p>Type an abbreviation you found from the Excel file.</p></li><br />
<li><p>Press “Search”</p></li><br />
<li><p>This will give the descriptor information.</p></li><br />
<li><p>You can also click on this link to find what the abbreviation means</p><br />
<ol style="list-style-type: lower-alpha;"><br />
<li><p><u>http://www.talete.mi.it/products/dragon_molecular_descriptor_list.pdf</u></p></li></ol><br />
</li></ol><br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<div title="footer"><br />
<br />
<span style="background: #c0c0c0">4</span><br />
<br />
<br /><br />
<br />
<br />
<br />
</div></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=WEKA_Steps_for_Loading_Data&diff=103WEKA Steps for Loading Data2022-09-26T19:21:31Z<p>Sysadmin: Imported from text file</p>
<hr />
<div>__TOC__<br />
<br />
<div title="header"><br />
<br />
{|<br />
|width="33%"| <br /><br />
|width="33%"| <br /><br />
|width="33%"| <span style="background: #c0c0c0">0</span><br />
|}<br />
<br />
<br /><br />
<br />
<br />
<br />
</div><br />
Steps for Loading Data into WEKA<br />
<br />
<br /><br />
<br />
<br />
ARFF format consists of three parts: @RELATION, @ATTRIBUTE and @DATA.<br />
<br />
<br /><br />
<br />
<br />
@RELATION ''“SPACE”'' name<br />
<br />
@ATTRIBUTE ''“SPACE”'' descriptor name ''“SPACE”'' data type (numeric, nominal …)<br />
<br />
<span id="_GoBack"></span> @DATA: numbers (integer or real) or strings<br />
<br />
<br /><br />
<br />
<br />
General rules for ARFF file can be found here<br />
<br />
https://www.cs.waikato.ac.nz/ml/weka/arff.html<br />
<br />
(Search “ARFF File Format”)<br />
<br />
<br /><br />
<br />
<br />
In Excel:<br />
<br />
* File 1: the format in this file is three columns for: “@ATTRIBUTE”, the descriptors’ names, and “NUMERIC”<br />
** Open the txt data file in Excel. Make sure you are searching from “All Files”.<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_4ec5c9075dd9cb51.png|512x288px]] [[File:WEKA_Steps_for_Loading_Data_HTML_ae35e05dbfaa6f7c.gif|31x38px|Shape1]]<br />
<br />
''Figure <span style="background: #c0c0c0">1</span>, Opening your data file''<br />
<br />
* When '''Text Import Wizard''' prompts, choose '''Delimited and''' click on '''Next''' ('''step 1'''), check the '''Tab''' box ('''step 2''') and click on '''Finish'''. (Leave the rest as default unless it is necessary to change.)<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_d288ed8722be4fdc.png|393x300px]] [[File:WEKA_Steps_for_Loading_Data_HTML_369e56d90bf80141.gif|32x31px|Shape3]] [[File:WEKA_Steps_for_Loading_Data_HTML_bef7f76831abec8b.gif|14x32px|Shape2]]<br />
<br />
''Figure <span style="background: #c0c0c0">2</span>, Step 1''<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_cd41ce313e7118b1.png|379x288px]] [[File:WEKA_Steps_for_Loading_Data_HTML_859149fac76acd64.gif|30x19px|Shape5]] [[File:WEKA_Steps_for_Loading_Data_HTML_6092b35113c6658c.gif|19x33px|Shape4]]<br />
<br />
''Figure <span style="background: #c0c0c0">3</span>, Step 2''<br />
<br />
* Create three blank columns.<br />
* Copy the descriptors’ names (they are at the first row of your data file, normally) and paste them in the vertical form by using '''“Transpose”''' pasting option '''to the second column'''.<br />
* Make sure the '''cell format''' is '''Text''' before the next step.<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_5e70e797a5c2cba0.png|313x156px]] [[File:WEKA_Steps_for_Loading_Data_HTML_634c3e41e8a46d53.gif|59x34px|Shape6]]<br />
<br />
''Figure <span style="background: #c0c0c0">4</span>, Cell format''<br />
<br />
* For the first and third columns: make equal number of rows as the second column has for “@ATTRIBUTE” and “NUMERIC”, respectively.<br />
* Example:<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_e8fa2497f6fc4487.png|198x337px]]<br />
<br />
''Figure <span style="background: #c0c0c0">5</span>, the three columns''<br />
<br />
<br /><br />
<br />
<br />
* File 2: Here we separate the @DATA part (only numbers) of data.<br />
** Keep only the numbers needed and delete everything else.<br />
** Save it in '''CSV (comma delimited)''' format. ('''Click YES''' when asks “Some features in your workbook … Do you want to keep using that format?”)<br />
<br />
<br /><br />
<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_d4bc2328d2c3083f.png|279x64px]] [[File:WEKA_Steps_for_Loading_Data_HTML_4fc01181394c3a93.gif|43x28px|Shape7]]<br />
<br />
''Figure <span style="background: #c0c0c0">6</span>, Save as CSV''<br />
<br />
<br /><br />
<br />
<br />
In Notepad++:<br />
<br />
* File A:<br />
** In the first row, create two columns: @RELATION and a title for the relation name (which are just separated by a space).<br />
** ''(For aesthetics:'' leave the second row blank.)<br />
** Copy the three columns in File 1 of Excel and paste into row3.<br />
** Example:<br />
<br />
[[File:WEKA_Steps_for_Loading_Data_HTML_1f0af9e857ead597.png|247x204px]]<br />
<br />
''Figure <span style="background: #c0c0c0">7</span>, start of the File A''<br />
<br />
* Depending on your data, the last @ATTRIBUTE row will need to be the response variable AKA the thing you are trying to predict. (CATEGORICAL may require this, NUMERIC may not)<br />
* After all ATTRIBUTE information has been pasted leave 1 row blank ''(For aesthetics'')<br />
* After blank row Type “@DATA”.<br />
* ''(For aesthetics:'' For the next row after “@DATA”, leave it blank.)<br />
* Example: [[File:WEKA_Steps_for_Loading_Data_HTML_ca4125bd73bd6170.png|460x172px]]<br />
<br />
''Figure <span style="background: #c0c0c0">8</span>, middle of File A''<br />
<br />
<br /><br />
<br />
<br />
* File B:<br />
** Open File 2 from Notepad++ (and you should see the data are separated by commas).<br />
** Example: [[File:WEKA_Steps_for_Loading_Data_HTML_964b046e1656a7f.png|555x225px]]<br />
<br />
''Figure <span style="background: #c0c0c0">9</span>, File 2 open with NotePad++''<br />
<br />
* Copy and paste all the data to File A.<br />
<br />
<br /><br />
<br />
<br />
* Already returned to File A:<br />
** Save the file as “arff” format (by adding “.arff” at the end of the file name).<br />
<br />
<br /><br />
<br />
<br />
'''The file is ready to open and run in WEKA (yay).'''<br />
<br />
PS: (1) “%” would give you error if included in the descriptors’ names, specifically such as '''“%N” or “%O”''', after “@ATTRIBUTE”. The error would occur because whatever after “%” is considered as comment, then in WEKA, it would be interpreted as a missing information for the descriptor name and data type. Simply remove “%” would avoid errors. (2)Also make sure the number of descriptors matches with the number of data in the “@DATA” section. (If you have 10 lines of @ATTRIBUTE + descriptors’ names, there should have 10 numbers in each line in @DATA part in the Notepad++.<br />
<br />
<br /><br />
<br /></div>Sysadminhttps://wiki.nanobiodata.org/index.php?title=VASP_and_VMD_on_CCAST&diff=102VASP and VMD on CCAST2022-09-26T19:21:18Z<p>Sysadmin: Imported from text file</p>
<hr />
<div>__TOC__<br />
<br />
<br /><br />
<br />
<br />
{|<br />
|<br />
'''Login''' '''Photon''':<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
ssh [mailto:kilin@photon.chem.ndsu.nodak.edu <u>kilin@photon.chem.ndsu.nodak.edu</u>]<br />
<br />
cd /scratch001/Chem_476/CHEM676_2019/Meade/DIMER<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
'''Login''' '''Cori''':<br />
<br />
<br /><br />
| Have to be in Photon first then login to Cori<br />
|-<br />
|<br />
ssh -i ~/.ssh/nersc cori.nersc.gov<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
cd /global/cfs/cdirs/m1251/vasp/CHEM676/meade<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
'''Misc. Tools'''<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| cd foldername<br />
| lets you go into a folder<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| cd ..<br />
| Backs you out of folder you are in<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| cd -<br />
| Goes back to previous directory<br />
|-<br />
| cd ~<br />
| Goes to home directory<br />
|-<br />
| freenodes<br />
| in ccast<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| vi filename<br />
| Lets you go into file to edit<br />
|-<br />
| i<br />
| When in “vi” this initiates the editing<br />
|-<br />
| :wq<br />
| This lets you stop editing<br />
|-<br />
| “Ctrl+c” press on keyboard, not a typing thing<br />
| Lets you stop any current running process<br />
|-<br />
| pwd<br />
| Gives you current directory location<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| wc<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| ls<br />
| Lists what is in the current directory<br />
|-<br />
| ls -lt<br />
| Lists what is in the current directory sorting from newest to oldest<br />
|-<br />
| ls –lt W*<br />
| Lists everything that starts with a W sorting from newest to oldest. (Can be used for other letters as well)<br />
|-<br />
| ls -lisanh<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisanh filename<br />
| gives information on the specific file<br />
|-<br />
| ls -lisan<br />
| gives information on the files in whole directory<br />
|-<br />
| ls -lisan filename<br />
| gives information on the specific file<br />
|-<br />
| ls -a<br />
| shows all files, including hidden ones<br />
|-<br />
| babel –ixyz file.xyz –opdb file.pdb xyz to pdb<br />
|<br />
Using babel, converts the input file of .xyz to outputfile of .pdb<br />
<br />
That is where the –ixyz and –opdb comes from<br />
|-<br />
| “shift+alt+left click”<br />
| For slecting just comlumn or just row information you want exactly<br />
|-<br />
| egrep '163|166' OS_STRENGTH<br />
| grep technique<br />
|-<br />
| more filename<br />
| Lets you see what is in the file<br />
|-<br />
| more ~/bin/filename<br />
| Lets you see the file in a location not in your current directory.<br />
|-<br />
| more filename | grep ### or value | grep again if desired<br />
| shows file and grabs lines with same input you put after the grep command<br />
|-<br />
| cp /Location/filename .<br />
| Copies a file from a different directory location and input it into the current directory. Don’t forget the PERIOD!<br />
|-<br />
| P /Location/filename ./newfilename<br />
| Copies a file from a different directory location and input it into the current directory and rename it in one step<br />
|-<br />
| scp sourceUser:/sourceLocation/filename .<br />
| Takes file from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/* .<br />
| Takes ALL files from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp sourceUser:/sourceLocation/*.filetype .<br />
| Takes ALL files with that specific file type from different SERVER/ACCOUNT. You must be in the destination directory you want the files to go to when typing this line<br />
|-<br />
| scp * destinationUser:/destinationLocation<br />
| Sends all files from source directory to destination directory. Must be in source directory when typing this line<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/DIMabsorptionspec<br />
| Example of scp<br />
|-<br />
| sbatch corifilename<br />
| Lets you submit a file in cori using a corifile made by others<br />
|-<br />
| grep specificdetail filename<br />
| Grabs the specific detail from the file you chose<br />
|-<br />
| sort -n -r filename<br />
| sorts the file in ascending order<br />
|-<br />
| babel BondLength.xyz -O out.xyz --separate<br />
| THIS IS CORRECT WAY THE SEP WORD IS SPELLED INCORRECTLY BUT IS CORRECT FOR BABEL<br />
|-<br />
| awk '{ if($4 &gt;= 4.0000 &amp;&amp; $4&lt;=5.0000) print }' OS_STRENGTH1<br />
| lets you get lines of data that contain the range from 4-5 from column 4<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| users<br />
| lets you see what users are on<br />
|-<br />
| clear<br />
| lets you clear screen<br />
|-<br />
| who<br />
| who is on and more detail about them<br />
|-<br />
| w<br />
| all the people and differing details about them<br />
|-<br />
| id<br />
| lets me see what groups I am in<br />
|-<br />
| freenodes<br />
| lets you see if nodes are available or not<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6335cce9234bc4ef.png|439x225px]]<br />
|<br />
|-<br />
| pbsnodes –avSj | sort -n<br />
| shows available nodes, the | sorts it into order<br />
|-<br />
| pbsnodes –avSj | sort –n | more<br />
| does this in page format rather than all at once<br />
|-<br />
| module avail<br />
| what software is available<br />
|-<br />
| $HOME<br />
| lets you go to your login directory “home” directory<br />
|-<br />
|<br />
thunder.ccast.ndsu.edu<br />
<br />
meade.erickson<br />
| to get into putty<br />
|-<br />
| C:\Users\meade.erickson\Documents\OriginLab\User Files\<br />
| Origin themes location<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Explanations'''<br />
| <br /><br />
|-<br />
| fermi<br />
| fermi energy is referring to the energy difference between the highest and lowest occupied single-particle states. ([https://en.wikipedia.org/wiki/Fermi_energy <u>https://en.wikipedia.org/wiki/Fermi_energy</u>])<br />
|-<br />
| '''Explanations Input Files:'''<br />
| <br /><br />
|-<br />
| INCAR<br />
| It is the central input file of VASP. It determines &quot;what to do and how to do it&quot;. ([https://www.vasp.at/wiki/index.php/INCAR <u>https://www.vasp.at/wiki/index.php/INCAR</u>])<br />
|-<br />
| POSCAR<br />
| <br /><br />
|-<br />
| POTCAR<br />
| <br /><br />
|-<br />
| KPOINTS<br />
| <br /><br />
|-<br />
| '''Explanations Output Files:'''<br />
| <br /><br />
|-<br />
| CONTCAR<br />
|<br />
Contains actual coordinates, velocities, and predictor corrector coordinates. 1<sup>st</sup> block is lattice parameters and atom coordinates. 2<sup>nd</sup> block Initial Velocites for atoms. 3<sup>rd</sup> block Predictor-corrector coordinates.<br />
<br />
Written at end of each job. Used as the POSCAR file for next MD job.<br />
|-<br />
| WAVECAR<br />
| Provides wavefunctions. we usually use it for spectrum data ([https://www.vasp.at/wiki/index.php/WAVECAR <u>https://www.vasp.at/wiki/index.php/WAVECAR</u>])<br />
|-<br />
| OS_STRENGTH<br />
| Information about the OS_STRENGTH<br />
|-<br />
|<br />
<br /><br />
<br /><br />
<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8bf47c7304419724.png|354x154px]]<br />
|<br />
From Wikipedia<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
In spectroscopy, '''oscillator strength''' is a dimensionless quantity that expresses the probability of [https://en.wikipedia.org/wiki/Absorption_(electromagnetic_radiation) <u>absorption</u>] or [https://en.wikipedia.org/wiki/Emission_spectrum <u>emission</u>] of [https://en.wikipedia.org/wiki/Electromagnetic_radiation <u>electromagnetic radiation</u>] in transitions between [https://en.wikipedia.org/wiki/Energy_level <u>energy levels</u>] of an atom or molecule<sup>[</sup>[https://en.wikipedia.org/wiki/Wikipedia:Accuracy_dispute#Disputed_statement <sup>''<u>dubious</u>''</sup>]<sup>'' – ''</sup>[https://en.wikipedia.org/wiki/Talk:Oscillator_strength#Wrong_definition_oscillator_strength <sup>''<u>discuss</u>''</sup>]<sup>]</sup>.<sup><u>[https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Demtr%C3%B6der2003-1 [1]][https://en.wikipedia.org/wiki/Oscillator_strength#cite_note-Robinson1996-2 [2]]</u></sup> The oscillator strength can be thought of as the ratio between the quantum mechanical transition rate and the classical absorption/emission rate of a single electron oscillator with the same frequency as the transition.<br />
|-<br />
| POTIM (in INCAR file)<br />
| timestep in femtoseconds. So POTIM=1 is 1 femtosecod<br />
|-<br />
| '''Density of States'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| To get DOS you need to run the correct calculations in VASP using 4 files. INCAR, POSCAR, POTCAR, OUTCAR<br />
| <br /><br />
|-<br />
| Once you get output from your submitted information to cori or photon you can follow these steps to get Density of states<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| Grabs the number of bands (we need this for later) (STATES)<br />
|-<br />
| vi input_overlap<br />
| editing input_overlap<br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| Number of Bands from the grep NBA OUTCAR line<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| quitting editing input_overlap<br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| getting energy pop from different location<br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| wc STATES<br />
| <br /><br />
|-<br />
| grep NELE OUTCAR<br />
| record number of electrons<br />
|-<br />
| grep NIO OUTCAR<br />
| record number of ions<br />
|-<br />
| grep E-fer OUTCAR<br />
| record the three values and their names. If this line doesn’t work the try next line)<br />
|-<br />
| grep ferm OUTCAR | tail –n 1<br />
| record the three values and their names<br />
|-<br />
| ~/bin/DRESS_DOS_NORM<br />
| Executes the DOS code from different file location<br />
|-<br />
|<br />
Number of states from grep NBA OUTCAR<br />
<br />
<br /><br />
| grep NBA OUTCAR line<br />
|-<br />
| -20<br />
| <br /><br />
|-<br />
| 20<br />
| <br /><br />
|-<br />
|<br />
E-fermi info from grep E_fermi<br />
<br />
<br /><br />
|<br />
To literally line up DOS and STATES, you may need to type in 0 for both fermi and alpha+beta.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
A more standard way is to use the middle point energy between HOMO and LUMO i.e. (E_HOMO+E_LUMO)/2 for fermi and 0 for alpha+beta. In this way, the mid point of DOS is at zero, anything below zero is occupied and above zero is unoccupied.<br />
|-<br />
|<br />
alpha+beta info from grep E_fermi<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| .01<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
|<br />
Number of ions from grep NIO OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
Number of electrons from grep NELE OUTCAR<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| 3<br />
| <br /><br />
|-<br />
|<br />
gnuplot ~/bin/gnuprog_DOS.fermi<br />
<br />
<br /><br />
| Plotting the DOS (This is done in NERSC or PHOTON)<br />
|-<br />
|<br />
ps2pdfwr DOS.ps<br />
<br />
<br /><br />
| Converting DOS file from .ps to .pdf<br />
|-<br />
|<br />
*** DISCLAIMER, DON”T HAVE TO USE THISBOX BUT CAN IF DESIRED For better box* this can be one big command or you can split it up into pieces. ( anything before a colon is one command) THIS IS DONE IN PHOTON*<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set out MEADESDOS.ps MAY NOT NEED THIS NOT SURE YET<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
unset key<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
plot [-6:6.6] 'DOS.fermi' using ($1-2.2617):5 w i title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-2.2617):4 w l title &quot;Dimer Density of States&quot; ls 4;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set tics font &quot;,30&quot;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set ytics offset 0,0,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set xtics offset -0.4,-1,0<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
CANT GET THIS TO GO TO OUTPUT FILE NOT SURE WHY<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.518567):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.518567):4 w l title &quot;Dimer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Dimer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
set style line 4 lw 4 lc -1; unset key; plot [-6.1:6.1] 'DOS.fermi' using ($1-0.173350):5 w i lw 2 title &quot;Electron Occupation&quot;, 'DOS.fermi' u ($1-0.17335):4 w l title &quot;Wet Monomer Density of States&quot; ls 4; set xlabel 'Orbital Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Density of States' font &quot;,30&quot; offset -1,0,0; set title &quot;Density of States Wet Monomer Molecule&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|<br />
I am initially specifying a style that I will be using. In this case chose style 4 and am editing it to what I want.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
lw stands for linwidth, lc line color, not sure where the library is but lc -1 is black, 0=red I think and find out other through trial and error<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
lw is just multiplyer, so lw 4 is 4 times larger than default.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
unset key gets rid of key,<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
[#,#] is the range of my x axis graph.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
($#-###): $# is the column I want to take data from which is being subtracted by the amount of shift I am giving to the data (we want data to be centered at 0 sometimes and so for DOS we take the middle between the start of non occupied plus the end of occupied and divide by 2 to get our shift. (Notoccupied + occupied)/2 ) and the number after the “:” is the column I am correlating the X and Y axis with. So when we plot stuff we need to specify what is the x and what is the y axis.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
not sure exactly what the “w i” portion does but it helps us change from points to lines and then is what fills in the red portion of our graph. (The “i” stands for impulses. Add an extra “lw 2’ to increase the line width as shown in green text.) “w l” does the same thing but DOESN”T fill out the the electron density portion.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
xlabel and y labels are the axis titles and offsets help position them due to the changing sizes of the axes and tics.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Tics are the numbers on the Y and x axis so ytics offsets them and xtics does the horizontal axis change.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Absorption Spectrum - Transition Energy eV – Wavelength nm'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
|<br />
copying INCAR-gm into the directory as long as you are in it<br />
<br />
MAKE SURE<br />
<br />
LWAVE= .TRUE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori07:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec/TEST<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Or make sure the INCAR has this Ag+Carboxyl<br />
<br />
#ISIF=7<br />
<br />
#PREC = HIGH<br />
<br />
ENCUT= 300<br />
<br />
IBRION =3<br />
<br />
POTIM=0.2<br />
<br />
NSW = 30000<br />
<br />
ALGO = NORMAL #48<br />
<br />
ISMEAR = 0<br />
<br />
SIGMA = 0.01<br />
<br />
LREAL = A<br />
<br />
#ISTART = 0<br />
<br />
LWAVE = .TRUE.<br />
<br />
LCHARG = .FALSE.<br />
<br />
EDIFF=1E-6<br />
<br />
EDIFFG=-0.05<br />
<br />
LVTOT = .FALSE.<br />
<br />
#LVDW=.TRUE.<br />
<br />
#SPRING = -5<br />
<br />
#LCLIMB = .TRUE.<br />
<br />
#IMAGES = 7<br />
<br />
#IOPT=7<br />
<br />
#ICHAIN=0<br />
<br />
NELM=40<br />
<br />
#ISPIN=2<br />
<br />
ISYM=0<br />
<br />
NPAR=4<br />
<br />
<br /><br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|345x215px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| It is necessary to have the ‘spectrum’ file made which comes from a healthy WAVECAR which is used to generate OS_STRENGTH. After OS_STRENGHT we make the spectrum ~/bin/SPECTRUM_imp3 file.<br />
| <br /><br />
|-<br />
| Assuming you have conducted a calculation which generated a healthy WAVECAR file.<br />
| <br /><br />
|-<br />
| If healthy WAVECAR exists then…<br />
| <br /><br />
|-<br />
| grep NBA OUTCAR<br />
| record Number of bands<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
|<br />
1<br />
<br />
Number of Bands<br />
<br />
1<br />
| 36740210<br />
|-<br />
|<br />
‘esc’ key<br />
<br />
:wq<br />
| <br /><br />
|-<br />
| ~/bin/extract_energy_pop.exe<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| <br /><br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| ./osc_str.exe<br />
| This makes the OS_STRENGTH (oscillator strength)<br />
|-<br />
| grep '2.0000 0.0000' OS_STRENGTH &gt; OS_STRENGTHvc<br />
|<br />
Takes the orbital pairs from 2.0000 to 0.0000 into new file.<br />
<br />
<br /><br />
|-<br />
| cp OS_STRENGTH OS_STRENGTH1<br />
| making sure I have original OS_STRENGTH<br />
|-<br />
| cp OS_STRENGTHvc OS_STRENGTH<br />
| making sure I have file OS_STRENGvc kept for 2.0000 – 0.0000 but it will be our OS_STRENGTH<br />
|-<br />
| wc OS_STRENGTH<br />
| Record First number<br />
|-<br />
| ~/bin/SPECTRUM_imp3<br />
| <br /><br />
|-<br />
|<br />
Number from OS_STERNEGTH<br />
<br />
.1<br />
<br />
10<br />
<br />
.001<br />
<br />
HOMO<br />
|<br />
Should be able to find HOMO from STATES file (or energy_pop)<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Can change the “.001” to “0.1 or 0.01” to smooth out the line<br />
|-<br />
| a ‘spectrum’ file should have been made. This holds all the spectrum data. The spectrum can be visualized using these commands in Cori. If you want to visualize in photon then use the gnuplot information below<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE<br />
|<br />
IN CORI YOU CAN USE THESE<br />
<br />
THEN SCP them to photon, then the desktop!<br />
|-<br />
| gnuplot ~/bin/gnuprog_SPE_nm_200_800<br />
|<br />
|-<br />
| ps2pdfwr SPE.ps<br />
|<br />
|-<br />
| ps2pdfwr SPE_nm.ps<br />
|<br />
|-<br />
| scp * kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/COOL_INTERMED_Mead/absorptionSpec<br />
| example make sure you are in the directory of the files you want to send. Send it to a premade directory in photon<br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| scp sourceUser:/sourceLocation/*.pdf .<br />
| This is done while in the desktop directory so on your desktop you will see all the pdf files your just generated in desktop<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
FOR PHOTON USE THESE STEPS<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
We are plotting the information of spectrum to get the spectrum in wavelength of nanometers nm and transition energy in electron volts eV<br />
| <br /><br />
|-<br />
| more spectrum<br />
|<br />
(Be sure to be in correct directory)<br />
<br />
Column 1 is the spectrum in wavelength of nm<br />
<br />
column 2 is the spectrum in transition energy of eV<br />
<br />
column 3 is absorbance/absorption in a.u. (absoption units). Absorbance goes with wavelength, absorption goes with transition energy<br />
<br />
column 4 idk what this is.<br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| plot ‘spectrum’ 2:3<br />
| This gives you transition energy in x axis and absorption in a.u for yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [0:10] 'spectrum' using 2:3 w l title &quot;Transition Energy&quot; ls 4; set xlabel 'Transition Energy, eV' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorption a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| This is transition energy in x axis and absorption in yaxis.<br />
|-<br />
| set style line 4 lw 4 lc 3; unset key; plot [200:800] 'spectrum' using 1:3 w l title &quot;Absorption Spectrum&quot; ls 4; set xlabel 'Wavelength, nm' font &quot;,30&quot; offset 0,-2,0; set ylabel 'absorbance, a.u' font &quot;,30&quot; offset -1,0,0; set title &quot;Absorption Spectrum Dimer&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
| nm in x axis and absorbance in yaxis<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''UV-Vis Absorption Spectrum'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
You need a POSCAR file from a previous calculation. SO if you have done a calculation this process can be followed.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
We did one calculation already but we need to do a “geometry optimization” for the next one.<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| create an absorption directory where ever you would like and go into this directory<br />
| <br /><br />
|-<br />
| cp ~/bin/INCAR-gm<br />
| copying INCAR-gm into the directory as long as you are in it<br />
|-<br />
| cp filelocation/CONTCAR ./POSCAR<br />
| Changing CONTCAR to POSCAR<br />
|-<br />
| cp filelocation/POTCAR .<br />
| <br /><br />
|-<br />
| cp filelocation/KPOINTS .<br />
| <br /><br />
|-<br />
| mv INCAR-gm INCAR<br />
| changing file name from INCAR-gm to INCAR<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| cp ~/bin/cori-regular.sh .<br />
| this is our job submission file<br />
|-<br />
| vi cori-regular.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b3108944fbb5d2b.png|352x219px]]<br />
|<br />
Make it Look like this.<br />
<br />
While in the screen press “I” to insert and edit the file.<br />
<br />
When done editing it press “:wq” to save and quit the file.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
If you don’t like the edits you made you can press “:q!” to exit the file without saving it.<br />
<br />
<br /><br />
|-<br />
| sbatch cori-regular.sh<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| '''Generating CANDIDATES file for Photofragmentation'''<br />
| <br /><br />
|-<br />
| This is after calculations have been ran for MD???<br />
|<br />
Watch this first‼ Part 2 of Dr.Kilins video for how to generate the CANDIDATES file.<br />
<br />
[https://www.youtube.com/watch?v=NIrs1yXJj1Y&feature=youtu.be <u>https://www.youtube.com/watch?v=NIrs1yXJj1Y&amp;feature=youtu.be</u>]<br />
|-<br />
| We want to get information from the WAVECAR that can help us determine the “electron hopping” so we have to find the OS_STRENGTH (oscillator strength) that is high. Since higher OS_Strength the higher the probability that given pair of orbitals is excited. STATES file should have been generated before I think.<br />
|<br />
Watch this second‼ Part 1 of Dr. Kilins video for what to do after CANDIDATES file was generated and explains what is in it.<br />
<br />
[https://www.youtube.com/watch?v=A8BXT-FnuxM&t=212s <u>https://www.youtube.com/watch?v=A8BXT-FnuxM&amp;t=212s</u>]<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| grep –A 700 fermi OUTCAR | tail –n 700 &gt; STATES1<br />
| Fermi information from OUTCAR into a file called STATES 1. Not sure what the –A or 700 mean. But the tail is choosing a specific section of the data, the “tail” end of the chosen information from the OUTCAR file.<br />
|-<br />
| more STATES1<br />
| Determine HOMO and LUMO, HOMO is the highest orbital with a 2, LUMO is the first 0. In my case HOMO is 112 and LUMO is 113<br />
|-<br />
| NOT SURE HOW HE GENERATED STATES FILE<br />
| <br /><br />
|-<br />
| head STATES<br />
| Record the lowest number, in my case it is 81 for DIMER<br />
|-<br />
| tail STATES<br />
| Record the Highest number, in my case it is 160 for DIMER<br />
|-<br />
| vi input_overlap<br />
| <br /><br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| 160<br />
| <br /><br />
|-<br />
| 1<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp energy_pop STATES<br />
| <br /><br />
|-<br />
| more ~/bin/ADVICE.spectrum<br />
| This will make spectrum. This will show 3 lines of code. Literally copy one line and press enter starting with the “module swap PrgEnv-intel PrgEnv-gnu”<br />
|-<br />
| module swap PrgEnv-intel PrgEnv-gnu<br />
| <br /><br />
|-<br />
| cp ~/bin/OS/2020/*.exe .<br />
| copying templates and inputting them into the directory you are in<br />
|-<br />
| ./osc_str_CHEM676.exe<br />
| This reads binary WAVECAR file, Reads input_overlap, creates Oscillator strength<br />
|-<br />
| ls –lt | head<br />
| This should show a fresh OS_STRENGTH was made<br />
|-<br />
| grep ‘2.0000 0.0000’ OS_STRENGTH &gt; OS_NEW<br />
| In the OS_STRENGTH file we will see lines of no transition from occupied to unnoccupied. What we want to see is the lines that have an occupied originally (2.0000) and then are unoccupied (0.0000). This shows excitation. The lines that meet this requirement are then inputted into OS_NEW<br />
|-<br />
| head OS_NEW<br />
| This should show a range of orbitals that have been occupied and are now unoccupied. With the differing orbitals involved. Columns 1 and 2 are the initial and final orbitals that are involved with the “hopping” of electrons. The 3rd column is oscillator Strength. 4th Column is Transition energy.<br />
|-<br />
| mv OS_NEW OS_STRENGTH<br />
| We don’t need all OS_STRENGTH data, just the stuff we “grepped” in OS_NEW. This overrides OS_STRENGTH with information from OS_NEW.<br />
|-<br />
| wc OS_STRENGTH<br />
| Word count: The 1st number shows number of lines present in file. 2<sup>nd</sup> number of words present in file. 3<sup>rd</sup> column shows number of characters present in file. 4<sup>th</sup> number is the file name which is given as an argument.<br />
|-<br />
| head –n 30 OS_STRENGTH<br />
| In this particular case we only want the top 30 rather than the original value the first column gave was. In my case it was about 1536.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4<br />
| To sort in ascending order of transition energy. 4 is the 4<sup>th</sup> column. I don’t know what –gk means.<br />
|-<br />
| head –n 30 OS_STRENGTH | sort –gk 4 &gt; CANDIDATES<br />
| Inputs this information into CANDIDATES file<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Orbital Visualization'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| I don’t have all information available here, for example I personally didn’t make the PARCHG files.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
Kilins Guidance to make PARCHG files… I think<br />
<br />
Dear Meade,<br />
<br />
below are steps I did for creating orbitals files on nersc.<br />
<br />
I will let you know when they are ready and instruct on how to visualize.<br />
<br />
slides are good, under given limits.<br />
<br />
In the given time, you may add conclusions as you see them..<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; mkdir ORBITALS<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp WAVECAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp CONTCAR ORBITALS/POSCAR<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp POTCAR ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp *.sh ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER&gt; cp KPOINTS ORBITALS/<br />
<br />
&gt; cd ORBITALS/<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; grep NBA ../OUTCAR<br />
<br />
k-points NKPTS = 1 k-points in BZ NKDIM = 1 number of bands NBANDS= 160<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; cp ~/bin/INCAR/INCAR-pc ./INCAR<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; tail INCAR<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
IBRION=2 #conjugate-gradient algorithm used to relax ions (bad<br />
<br />
NSW=0 #number of ionic steps<br />
<br />
POTIM= .2 #time step in fs<br />
<br />
EDIFFG=-0.001 #minimum energy difference between ionic iterations<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
#EINT= -4.4 -0.5<br />
<br />
EINT= -6 2 <br />
<br />
LVTOT = .TRUE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
kilin@cori11:/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/ORBITALS&gt; sbatch cori_debug.sh <br />
<br />
Submitted batch job 32413471<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
many thanks for the new version!<br />
<br />
The main question in your email is orbitals with VASP:<br />
<br />
a. generate or use premade healthy WAVECAR <br />
<br />
(it is an analog of .chk in gaussian)<br />
<br />
b. in a new directory, use template of the INCAR for orbitals and run vasp once again.<br />
<br />
the templates are named ~/bin/INCAR/INCAR_pc and similar names.<br />
<br />
(it is an analog of the cubegen procedure in gaussian)<br />
<br />
c. upon successful run, there appear files nammed PARCHG.XXXX.ALLK, where XXXX stands for the index of orbital, e.g. PARCHG..01112.ALLK<br />
<br />
(those files are analogs of the .cube files in gaussian. note that vasp DOES not generate literal .cube files!)<br />
<br />
d. download these files and open then in the VMD - the same software you have used for the trajectories.<br />
<br />
I will have a look on the slides and let you nko if any quicl changes are needed or we can go ahead with this version<br />
<br />
Best wishes,<br />
<br />
Dmitri<br />
<br />
<br /><br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Step 2. Changing PARCHG Files to cube files (For Orbital Visualization)<br />
| From what I have seen we need cube files to show in guassview AND VMD<br />
|-<br />
| Once obtained PARCH.####.ALLK Files then use following command in PHOTON to convert the orbitals you care about to cube!<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| ~/vtstools/chg2cube.pl filename<br />
| <br /><br />
|-<br />
| 3<br />
| selecting PARCHG<br />
|-<br />
| 1 6 7 8<br />
| (These are the atomic numbers for the atoms in your system. In my case I have Hydrogen, Nitrogen, Oxygen, and Carbon<br />
|-<br />
| Your cube file is then generated<br />
| <br /><br />
|-<br />
| SEND cube files, POSCAR (OR CONTCAR), CHGCAR FILES TO DESKTOP<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| Step 3. Using VMD to generate orbitals<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| Open VMD<br />
| <br /><br />
|-<br />
|<br />
In the VMD Main window go to “File” “new Molecule”<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Upload your<br />
<br />
1.POSCAR (OR POSSIBLY CONTCAR): By selecting VASP_POSCAR under the “Determine file type:”<br />
<br />
2. CHGCAR: By selecting VASP_CHGCAR under the “Determine file type:”<br />
<br />
3. USE THIS ONE PARCHG: By selecting VASP_PARCHG under the “Determine file type:” (Even though the file ends in .cube we are classifying it as PARCHG)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b0283f3c67e04e7d.png|219x134px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_eae9e77b87b8627b.png|222x103px]]<br />
|-<br />
|<br />
After all three are uploaded,<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
in the VMD Main window go to “Graphics” and then “Representations”<br />
| <br /><br />
|-<br />
|<br />
In the Graphical Representations window follow these three examples.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
(Change the file you are using by the dropdown menu of “selected Molecule”)<br />
<br />
<br /><br />
|<br />
The red lettering means that PARCHG Isosurface isn’t being shown. You can double click on the red lettering to make it black and showing on the OpenGL Display Window. Double click again to revert to black.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Parchrg can be shown and can be visualized in conjungtion with the CHGCAR file<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_e7a558f4dc169c4e.png|121x260px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6b15ffb6b911b765.png|124x266px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3734231f026ab64e.png|124x265px]]<br />
|<br />
|-<br />
|<br />
-You can change the iso value in CHGCAR to get varying orbital visualizations.<br />
<br />
-Play with it.<br />
<br />
-Change the color by changing the ColorID to different ones.<br />
<br />
-(Range in CHGCAR doesn’t matter)<br />
|<br />
For Monomer Project Show<br />
<br />
all index 52 47 29 44 38 28 18 56 64<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
H52<br />
<br />
H47<br />
<br />
H29<br />
<br />
H44<br />
<br />
H38<br />
<br />
H28<br />
<br />
H18<br />
<br />
H56<br />
<br />
H64<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c5f67d67a9a89fa7.png|355x180px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d4c6c68549d8e025.png|357x209px]]<br />
| <br /><br />
|-<br />
| To get rid of opaque water molecuels make both of these in “Graphics” “Materials” 123 is for faded bods and 124 is for the orbitals<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_8d905765b5183c3.png|142x229px]] [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_449e5be8bb4a3541.png|139x222px]]<br />
|<br />
|-<br />
| For my Monomer_H2O first frame hide these atoms if you don’t want in view<br />
| <br /><br />
|-<br />
|<br />
<br /><br />
<br /><br />
<br />
<br />
O108<br />
<br />
O99<br />
<br />
O94<br />
<br />
O101<br />
<br />
O81<br />
<br />
O91<br />
<br />
O90<br />
<br />
O86<br />
<br />
O82<br />
<br />
O87<br />
<br />
O79<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
H73<br />
<br />
H45<br />
<br />
H62<br />
<br />
H51<br />
<br />
H58<br />
<br />
H50<br />
<br />
H48<br />
<br />
H14<br />
<br />
H22<br />
<br />
H36<br />
<br />
H42<br />
<br />
H31<br />
<br />
H20<br />
<br />
H23<br />
<br />
H35<br />
<br />
H30<br />
<br />
H54<br />
<br />
H59<br />
<br />
H25<br />
|<br />
all not index ### ### ### ###...<br />
<br />
all not index 108 99 94 101 81 91 90 86 82 87 79 73 45 62 51 58 50 48 14 22 36 42 31 20 23 35 30 54 59 25<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
backbone ithink<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
index 97 103 11 9 10 13 12 96 1 84 78 3 4 100 77 0 8 2 5 6 7<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''These are for the figure first frame for still seeing the water atoms but hazy'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
H22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
O90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
| 22 14 26 19 30 35 15 17 60 69 73 62 45 34 40 16 21 37 27 55 53 49 24 32 43 39 33 67 70 71 65 75 76 66 72 68 63 59 50 51 58 54 48 20 47 23 36 42 31 25 74 61 57 41 46 90 86 91 99 94 104 106 111 109 110 107 92 85 95 102 88 80 89 108 105 83 87 82 81 90 101 79 93<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Index for Diagram for pics Middle Frame for H20 molecules'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
H63<br />
<br />
H33<br />
<br />
H17<br />
<br />
H21<br />
<br />
60<br />
<br />
68<br />
<br />
42<br />
<br />
38<br />
<br />
34<br />
<br />
50<br />
<br />
72<br />
<br />
73<br />
<br />
16<br />
<br />
61<br />
<br />
55<br />
<br />
71<br />
<br />
59<br />
<br />
69<br />
<br />
40<br />
<br />
14<br />
<br />
39<br />
<br />
74<br />
<br />
67<br />
<br />
53<br />
<br />
49<br />
<br />
41<br />
<br />
52<br />
<br />
27<br />
<br />
15<br />
<br />
31<br />
<br />
23<br />
<br />
54<br />
<br />
62<br />
<br />
32<br />
<br />
64<br />
<br />
30<br />
<br />
26<br />
<br />
57<br />
<br />
37<br />
<br />
51<br />
<br />
58<br />
<br />
70<br />
<br />
43<br />
<br />
25<br />
<br />
35<br />
<br />
46<br />
<br />
45<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|<br />
O80<br />
<br />
87<br />
<br />
99<br />
<br />
81<br />
<br />
88<br />
<br />
105<br />
<br />
104<br />
<br />
91<br />
<br />
90<br />
<br />
106<br />
<br />
97<br />
<br />
108<br />
<br />
102<br />
<br />
89<br />
<br />
93<br />
<br />
101<br />
<br />
92<br />
<br />
107<br />
<br />
78<br />
<br />
109<br />
<br />
86<br />
<br />
83<br />
<br />
94<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
63 33 17 21 60 68 42 38 34 50 72 73 16 61 55 71 59 69 40 14 39 74 67 53 49 41 52 27 15 31 23 54 62 32 64 30 26 57 37 51 58 70 43 25 35 46 45 80 87 99 81 88 105 104 91 90 106 97 108 102 89 93 101 92 107 78 109 86 83 94<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_783e625ce0b8a321.png|182x247px]]<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Preparation of TDESMD Orbital Files and Job Submission'''<br />
| <br /><br />
|-<br />
| mkdir TDESMD_111_114<br />
| We will now use the information of the CANDIDATES file and create new folder for each scenario. You create a file titled using the orbitals involved. In my case the first scenario is with orbitals 111 and 114<br />
|-<br />
| cd TDESMD_111_114<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/INCAR_EXCITED<br />
| Taking INCAR_EXCITED file from location. In my case it is here.<br />
|-<br />
| vi INCAR_EXCITED<br />
| Edit the INCAR_EXCITED file to make it look like<br />
|-<br />
| i<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_79a32df6f9f33bb0.png|303x201px]]<br /><br />
|-<br />
| <br /><br />
|<br />
Change FERWE=#### to match the total number of orbitals/bands. In this example we have a total number of 160 bands/orbitals.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The first segment of FERWE, “110*1” says we have the first 110 orbitals occupied. The second portion is orbital number 111 which we have listed as a “0” signifying this orbital is unnoccopied for our job submission. Then orbital number 112 is occupied with “1”. Then orbital number 113 is unoccupied with “0”. Then orbital number 114 is occupied with “1”. The last part of “46*0” is talking about the other orbitals not mentioned. These orbital should add up to 160. These numbers change with the differing orbitals designating in the CANDIDATES FILE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The main thing that is changing is the position of one occupied orbital and one unoccupied orbital when the compound is in ground state. In this example number 111 is usually occupied in ground state and we are moving the electron to 114 which is unoccupied in ground state.<br />
|-<br />
| <br /><br />
| ONE more example is as follows. With TDESMD_102_115<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_51e84877e4374ca6.png|300x199px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POSCAR ./CONTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/KPOINTS<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/POTCAR<br />
| <br /><br />
|-<br />
| cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_111_114/debug1.sh<br />
| <br /><br />
|-<br />
| sbatch debug1.sh<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Creating Videos from movieAll.xyz files and uploading them on youtube'''<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| scp –r TDESMD_###_### DestinationUser:/DestinationLocation<br />
|<br />
Starting in cori<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Using this to send files from cori to photon. We specifically want the movieALL.xyz files.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
This is sending the WHOLE directory and everything it contains<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
--OPEN A NEW TAB IN MOBA XTERM--<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Without logging into the account you should be able to see if you are on your current computer<br />
| <br /><br />
|-<br />
| ls<br />
| make sure you are in your computer directory. Usually you will see “DESKTOP” as an option<br />
|-<br />
| cd DESKTOP<br />
| <br /><br />
|-<br />
| mkdir TDESMD_###_###<br />
|<br />
In DESKTOP<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
generate TDESMD_###_### directories<br />
|-<br />
| scp kilin@photon.chem.ndsu.nodak.edu:/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_##_###/movieALL.xyz .<br />
| WHILE IN THE TDESMD DIRECTORY THAT YOU WANT TO BE THE DESTINATION take “movieALL.xyz” from photon and put into specific directory<br />
|-<br />
| <br /><br />
| Once the file is in your desktop you can go to your windows folders and open it to check if it is there.<br />
|-<br />
| In windows search bar<br />
| <br /><br />
|-<br />
| VMD<br />
|<br />
type vmd in windows search bar and open it.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
VMD will open with multiple screens.<br />
<br />
<br /><br />
|-<br />
| We will focus on this screen first to open all the extra screens we need. (VMD Main)<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_5f31d31c2c72c623.png|244x114px]]<br />
|-<br />
| Press “File”, “New Molecule,”<br />
| <br /><br />
|-<br />
| On the window that comes up (Molecule File Browser) Press “Browse” and find the file you would like. Find the file you want. In this example it is an xyz file.<br />
| <br /><br />
|-<br />
| Once A file is picked then press “Load” in the Molecule File Browser window.<br />
| <br /><br />
|-<br />
| Going back to VMD Main<br />
| <br /><br />
|-<br />
|<br />
Press “Graphics” and Representations to open a long slender screen.(VMD Main)<br />
<br />
<br /><br />
<br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c8cfeb66ca7723b0.png|225x487px]]<br />
|-<br />
| Press the “Create Rep” Button (Graphical Representation window)<br />
| <br /><br />
|-<br />
| With one of the “Reps” highlighted in green, in the drawing Method switch from Lines to VDW<br />
| <br /><br />
|-<br />
|<br />
Change the Sphere scale to 0.2.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Resolution can stay at default or can be raised if desired.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6ceb4d7e0c7c142a.png|157x340px]]<br />
|-<br />
|<br />
Click on the other line that still says “LINES” and in the Drawing Method change it to Dynamic Bonds<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Change Distance Cutoff to 1.6<br />
<br />
Bond Radius to 0.1<br />
<br />
And Resolution can stay at default or be increased<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_dba6c2b704e3401a.png|158x337px]]<br />
|-<br />
| Go back to VMD Main<br />
| <br /><br />
|-<br />
| Go to “Extensions,” “Visualizations,” and “Movie maker”<br />
| <br /><br />
|-<br />
| Go to the VMD Movie Generator window<br />
| <br /><br />
|-<br />
| In the box of “Name of movie” type the TDESMD_##_### you have.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|214x167px]]<br />
|-<br />
| “Set Working Directory“ choose the destination directory you want the movie to go to.<br />
| <br /><br />
|-<br />
| “Rotation Angle” I put it to 0 but you can put it to whatever.<br />
| <br /><br />
|-<br />
| “trajectory step size” 1<br />
| <br /><br />
|-<br />
| “Movie Duration” This depends on your frames from the (VMDMain) Usually 1000 frames is about 20-30 seconds if the speed (step) is set to 1. But By changing the speed of the video we should make the total video length to 30-45 seconds long.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b50ecc50051093d9.png|232x181px]]<br />
|-<br />
| Back to VMD Main window<br />
| <br /><br />
|-<br />
| Press “Mouse,” “Label,” “Bonds”<br />
| <br /><br />
|-<br />
| click on two neighboring atoms to get the atom name and bond length<br />
| <br /><br />
|-<br />
|<br />
IN VMD Main, go to “Graphics” and “Labels.” select “atoms” in the drop down menu on the “Labels” window.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Select all atoms by clicking on the top atom and then “SHIFT+left click” the bottom atom to select all atoms<br />
<br />
<br /><br />
| <br /><br />
|-<br />
|<br />
While all atoms are selected (highlighted in green). Go to “Properties” and in the “Format” box type %e%i.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
(The %e gives the element and the %i gives the index number)<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Some information can be presented by simply having your cursor over the %e%i and a yellow box will pop up for some info.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
You can move the lettering of each atom by moving the offset information.<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_46eab8f828aa40bc.png|215x201px]]<br />
|-<br />
| Click the global properties and change text size and text thickness. ( all of this can be see in the (VMD1.9.3 OpenGL Display) [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_260ebe13a2eda05.png|162x114px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3968014bc41ed6a4.png|224x212px]]<br />
|-<br />
| in VMD movie Generator press “make movie” and quickly in VMD main press the play button.<br />
| <br /><br />
|-<br />
|<br />
Once the video is done a popup should be produced asking where to find videomach. Here is the directory for my computer.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
C:\Program Files (x86)\VideoMach\videomach.exe<br />
<br />
<br /><br />
| <br /><br />
|-<br />
| Once you put in the file videomach.exe then videomach should show up.<br />
| <br /><br />
|-<br />
| Switch the file type to MP4<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_6a986dcfd4344618.png|374x27px]]<br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_da932791910b2a04.png|167x188px]]<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_d35497678abed954.png|155x364px]]<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| Polymer/Monomer Photodegradation: TDESMD 91 121 Bond Data<br />
|-<br />
| <br /><br />
|<br />
Bond lengths shown (white lettering) are chosen to see if this degradation simulation relates to the photo cleavage explained in this specified paper.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Angew. Chem. Int. Ed. 2015, 54, 1159 –1163<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The transition energy for this excitation is 0.24386664.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
From conformation labeled as &quot;DIMER&quot;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Acknowledgements: Dr. Dmitri Kilin<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Obtaining MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| We will be using python scripts that are premade. In PHOTON<br />
| <br /><br />
|-<br />
| go to the TDESMD_###_### directory you want to go to<br />
| <br /><br />
|-<br />
| in here make another directory called massspectrum<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| mkdir massspectrum<br />
| <br /><br />
|-<br />
| cd masssspectrum<br />
| <br /><br />
|-<br />
| cp /Location/movieALL.xyz<br />
| This should be from the TDESMD_###_### you are in.<br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieALL.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Record number of molecules converted.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| python3 script1n.py<br />
| <br /><br />
|-<br />
| enter number of molecules converted<br />
| <br /><br />
|-<br />
| <br /><br />
| a frequency file should be created<br />
|-<br />
| cp frequency.txt frequency1.txt<br />
| <br /><br />
|-<br />
| awk 'ORS=(!(NR%2)?&quot;\n&quot;:&quot; &quot;)' RS=&quot; &quot; frequency.txt &gt; file1.txt<br />
| <br /><br />
|-<br />
| cp rmovingfile.sh<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
|-<br />
| bash rmovingfile.sh<br />
| <br /><br />
|-<br />
| vi file1.txt<br />
| Edit the file by deleting everything that doesn’t relate to the numbers.<br />
|-<br />
| i<br />
| <br /><br />
|-<br />
| before [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_b3731e92878413d2.png|289x328px]]<br />
|<br />
after<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_c6aa114dd0a5843f.png|108x333px]]<br />
|-<br />
|<br />
IT IS NECESSARY TO MENTION THAT WHEN CREATING THE FRAGMENTS FILE AN EXTRA BLANK LINE WILL BE GENERATED. IT IS ESSENTIAL TO DELETE.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_2104ec24e9b14aae.png|258x363px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_36d7bfd4ff6b9e13.png|218x371px]]<br />
|-<br />
| “esc” key<br />
| <br /><br />
|-<br />
| :wq<br />
| <br /><br />
|-<br />
| cp file1.txt fragments<br />
| <br /><br />
|-<br />
| wc fragments<br />
|<br />
record the first number<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
in my case it is 21<br />
|-<br />
| ~/bin/DRESS_FRAGMENT_v2<br />
| <br /><br />
|-<br />
|<br />
21<br />
<br />
100<br />
<br />
300<br />
<br />
.1<br />
<br />
1<br />
<br />
62<br />
<br />
160<br />
<br />
<br /><br />
|<br />
number of states<br />
<br />
Min MASS (can change if desired)<br />
<br />
Max MASS (can change if desired)<br />
<br />
width of the line used<br />
<br />
magnitude (keep at one)<br />
<br />
number of atoms<br />
<br />
number of electrons<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''Plotting FERWE MASSPECTRUM'''<br />
| <br /><br />
|-<br />
| gnuplot<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| e in the directory of the file you want to plot, in this example we are using MASS_SPEC.norm as our file with the data we want.<br />
|-<br />
|<br />
set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity (a.u)' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Monomer H2O Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0;<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
To press the right mouse button and move mouse to preferred window size. Then left click<br />
| monomer h20<br />
|-<br />
| ressing p takes you to the previous zoom level, n to the next level, and u unzooms<br />
| <br /><br />
|-<br />
| set style line 4 lw 100 lc -1; unset key; plot &quot;MASS_SPEC.norm&quot; using 3:5 w l lw 3; set xlabel 'Molecular Weight, amu' font &quot;,30&quot; offset 0,-2,0; set ylabel 'Intensity' font &quot;,30&quot; offset -1,0,0; set title &quot;Mass Spectrum Dimer Multiple Cells&quot; font &quot;,30&quot; offset 0,0.5,0; set tics font &quot;,30&quot;; set ytics offset 0,0,0; set xtics offset -0.4,-1,0<br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| '''Obtaining Multiple Cell MASSPECTRUM Data from FERWE TDESMD Calculations'''<br />
| <br /><br />
|-<br />
| Make sure TDESMD CONTCAREXICTED and CONTCARGROUND files have been generated. Start in Cori go to Photon<br />
| <br /><br />
|-<br />
|<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test.sh<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ test_o.sh<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/ANA_v2.exe<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/input_ANA<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cp /global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/CONTCAREXICTED and GROUNDFILES<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
|<br />
Take these files from Cori and put them into photon.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
These two files are needed for running the scripts properly<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
These are the “important parts of the code”<br />
<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
input_ANA should have similar format to this<br />
<br />
Where the first 3 numbers are from POSCAR, the last 3 are the number of cells in the x,y,z axes respectively. This files was duplicated across Z axis.<br />
<br />
18.54438999999999<br />
<br />
14.45937999999995<br />
<br />
17.76793999999994<br />
<br />
1<br />
<br />
1<br />
<br />
10<br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exe<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
I used input_ANA<br />
|-<br />
|<br />
CHECK to see if OUTPUT has the correct number of atoms<br />
<br />
[[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_4a92367ce10afd50.png|624x64px]]<br />
| Single cell had 620 atoms, I multipled it by 10 since I wanted to duplicate 10 times. If this is good procede to next steps<br />
|-<br />
| vi test.sh<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_19d33c90b0cefb7a.png|278x215px]]<br />
| change the number to later stage in your trajectory(or the number where you think degradation had occurred. For example my total CONTCARGround and excited was about 191 files. I saw degradation occur later in the simulation so I chose the number 190. This changes with the simulations.<br />
|-<br />
| bash test.sh<br />
| this runs the test.sh file<br />
|-<br />
| bash test_o.sh<br />
| <br /><br />
|-<br />
| NOW LOOKING INTO PHOTON<br />
| <br /><br />
|-<br />
| cp all files into photon directory<br />
| <br /><br />
|-<br />
| cp /Location/script1n.py<br />
| /scratch001/Chem_476/CHEM676_2019/Meade/Pythonscript/script1n.py<br />
|-<br />
| babel –ixyz movieANA.xyz shot.xyz -m<br />
|<br />
Creating shot files. Each shot is a “frame” from the video.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Record number of molecules converted.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
-2055 molecules converted<br />
<br />
-2055 files output. The first is shot1.xyz<br />
|-<br />
| Follow regular mass spec steps from above. Rather than using MovieALL we are using movieANA.xyz.<br />
| <br /><br />
|-<br />
| <br /><br />
|<br />
photon<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
/scratch001/Chem_476/CHEM676_2019/Meade/DIMER/TDESMD_112_116/massspectrum/MultipleCellMassSpec<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cori<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
/global/cfs/cdirs/m1251/vasp/CHEM676/meade/DIMER/TDESMD_112_116/massspec/MultipleCellMassSpec/test<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| plummer code for renaming files<br />
| /global/project/projectdirs/m1251/vasp/CHEM676/meade/MONOMER_H2O/TDESMD_162_167/massspec/multiplemassspectest/plumoutput<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| '''How to read/get the fragments with their xyz coordinates'''<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
Go into the directory with the shot files with the multiple cells. Make sure that some files have this comment inside of them by opening a shot file that has name shot##n#.xyz<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
The comment should be.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
created by duplicating cells#<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| grep -E 'created by duplicating cells.* 17.' shot*<br />
|<br />
finds lines that have both “created by duplicating cells” and the number 17 in the same line.<br />
<br />
17 is the mass number we are looking for<br />
<br />
I am unsure what the “.*” does in this code<br />
<br />
make sure there is a space between the * and the 17<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Then open one of the xyz files listed.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Literally drag and copy the file information into a notepad document in your desktop.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| more shot291n10.xyz<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_9c48a58b9e24dc27.png|210x268px]]<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_7c3a89906ed25e30.png|180x270px]]<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| open the xyz file in Avogadro and save as mol file.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| the Avogadro file is correct visualization of the fragments. Mol file will have some defects so you will have to change those.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| Making Video from output of VASP<br />
| <br /><br />
|-<br />
|<br />
~/vtstools3/pos2xyz.pl CONTCAR<br />
<br />
cp CONTCAR.xyz INPUT.xyz<br />
<br />
~/bin/ANA_v2.exemore<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
cat OUTPUT.xyz &gt; movieANA.xyz<br />
<br />
<br /><br />
|<br />
Delete movieANA.xyz if you have it already.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
( This is to make sure the OUTPUT and initial part of the moveANA.xyz file is what it needs to be. In my situation I wanted to have 10 cells combined together.<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
I used input_ANA<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
| '''LAAMPS'''<br />
| <br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_796f3aade1e5b306.png|314x150px]]<br />
|<br />
testwater file:<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Epsolon is Inputs from something like DFT<br />
<br />
<br /><br />
<br /><br />
<br />
<br />
Coloumns:<br />
<br />
Atom ID<br />
<br />
Molecule ID<br />
<br />
Atom Type<br />
<br />
Q - charge<br />
<br />
X<br />
<br />
Y<br />
<br />
Z<br />
<br />
<br /><br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_f474cba5e65bdaab.png|306x66px]]<br />
| This infor can be put in input script if desired.<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_a9802b534de46367.png|624x180px]]<br />
|<br />
bond id<br />
<br />
bond tuype<br />
<br />
atom id that connects<br />
<br />
atom id that connects as well<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_3b00d172c3a15f2b.png|310x103px]]<br />
|<br />
Angle ID<br />
<br />
Angle Type 1 being single bond 2 being double, etc<br />
<br />
Item ID such as first atom O , 2<sup>nd</sup> and 3<sup>rd</sup> atom H<br />
|-<br />
| qsub submit.sh<br />
| submits the job using submit.sh<br />
|-<br />
| <span style="background: #101010">qstat -u meade.erickson</span><br />
| shoes me my jobs<br />
|-<br />
| <span style="background: #101010">chmod -777 lmp_mpi_12Dec18</span><br />
| modifies permission for this file<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| [meade.erickson@login0002 testwater]$ chmod -777 ~/bin/lmp_mpi_12Dec18<br />
| giving permission while in a different<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| topo readlammpsdata project.data<br />
| reads data file for vmd tk console<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| Inputting files, put in .dcd file then right click on it and “Load Data into Molecule” then select .data file.<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| [[File:VASP_VMD_LAMMPS_CCAST_LINUX_HTML_73b2551819922e26.png|287x164px]]<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| module avail<br />
| Available software in ccast<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| qdet job id<br />
| qdel ######<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| fragment 0 to 9 or fragment 14<br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
|<br />
|<br />
|-<br />
| <br /><br />
| <br /><br />
|}<br />
<br />
<br /><br />
<br /></div>Sysadmin