PertEDA

PertEDA is a Python package for basic data analysis of perturbed gene expression data. With a single line of code many statistical values are calculated and plots created for an easy overview.

PertEDA reposatory

https://bitbucket.org/sonnhammergrni/perteda/

Installation

Ensure Python and pip are installed. Use of virtual environments (venv) is recommended. Good instructions for this can be found on the internet. PertEDA is built with Python 3.12 and should work on surrounding versions.

On a Linux command line, install PertEDA using:

git clone git@ssh.bitbucket.org:sonnhammergrni/perteda.git
cd perteda
pip install .

Demo

There is an example script in the repository which showcases a mix of different data input and output options.

# cd pertEDA  # If you are not already in the repository
python3 eda_demo.py

Example 1: A pandas dataframe is used as input and no other options are specified. The output will be a dictionary with statistics to standard out (printed in the example) and the figures are shown.

Example 2: A csv file with gene expression where multiple genes are perturbed for every experiment is used as input. pert_design is therefore set to multiple. synthetic is set to True to be able to infer which experiments are replicates, as the experiment names are generated and uses the separator R to mark replicates. The output is specified without a file extension so both a json and a html file will be created. This is recommended for large datasets as the statistical measurements can continue to be accessed from the json as well as displayed neatly in the html.

Example 3: A csv file with gene expression where a single gene is perturbed per experiment is used as input. synthetic is set to True to be able to infer which experiments are replicates, as the experiment names are generated and uses the separator R to mark replicates. The output is specified with the extension .html to only create a html file.

Example 4: A csv file with gene expression with single perturbation, without any labels is used as data input. perts_df is set to a separate csv file, containing a perturbation matrix where perturbed genes are marked with a positive value, which will be used to determine replicates and which gene is perturbed per experiment. The output is specified with .png which will save all figures to a single png and the statistics in a json file.

Example 5: A csv file where the experiment and gene labels are matched to infer which gene is perturbed in which experiment. plot_interact is set to True so the output will be in the form of a Dash app. The app is opened by clicking the http link, and fully closed by interrupting the command (e.g. typing ctrl+c in a linux terminal).

Use in Python

It is possible to use PertEDA by importing as a package or by running pertEDA.py in the terminal. To use the function in a script:

from pertEDA import perteda

The possible inputs of the function are:

• data: pandas.DataFrame or csv file. Matrix with gene expression, typically on the form genes by experiments Options:
• perts_df: optional perturbation matrix as pandas.DataFrame or csv file, only used if pert_design is set to single
• norep: boolean option for when there are no replicates. Will limit some analyses, but runs the rest without errors from wrong assumptions.
• pert_design: option to change between designs with a single and multiple perturbed genes per experiment
  • single (default) - one gene perturbed per observation
  • multiple - one or more genes perturbed per observation
• out: option to write output to file, extension alternatives:
  • None (default) - will write to both json and html
  • json - will only write output to json, with figures encoded
  • html - will only write to html file and bypass json
  • png - will make one png file with all figures and write statistics to json file
• extra_info: optional csv file name with text or data to be displayed as a table in html output file
• direction: option to specify if genes or experiments are on the rows
  • None (default) - assumes largest direction are experiments
  • genes
  • experiments
• synthetic: boolean option for if the data labels replicates with R as a separator
• quick_calc: boolean option to skip SVD and correlation statistics which takes up the most time for large datasets
• plot_interact: boolean option to make a Dash app with interactive plots, will not save or output statistics or plot in any other way

Output

The statistics and figures can be saved in a variety of formats or viewed as an interactive Dash app. The statistics calculated are:

• Number of elements
• Number and percent of unique values
• Number and percent of zeros
• Number of genes
• Number of experiments
• Mean value
• Median value
• Max value
• Min value
• Variance
• Standard deviation
• Skewness
• Signal to noise ratio: mean over standard deviation
• Condition number
• Max, median and min of SVD
• Median rank and percent of perturbed gene: the percentage will never be 0 as it is the rank expressed as a percentage from 100/number of genes to 100 for easier comparison between datasets
• Times and percent lowest rank: the number of times and the percent of experiments where the perturbed gene was ranked lowest
• Times and percent highest rank: the number of times and the percent of experiments where the perturbed gene was ranked highest
• Median observation correlation: of all experiments
• Median replicate correlation: median correlation of experiments which are replicates of each other
• % replicate corr>median corr: the percentage of times where the correlation between two experiments are above the median overall correlation
• Median # of replicates
• Min # of replicates
• Max # of replicates

The plots created are a heatmap of all expression values, a histogram of the distribution of them, and two component PCAs over the genes and the experiments.

Interactive plots

The Dash app contains versions of the static plots which can be interacted with as long as the Flask server is running.

• Heatmap: select which experiments to display and whether to show all genes or only the ones perturbed in the experiments chosen
• Histogram: choose which range of values to focus on
• PCA over experiments: choose which experiment to highlight
• PCA over experiments animation: loops over experiments or replicates (most applicable for batched data where replicates and cells aren’t the same) and highlights one group at a time
• PCA over genes

Large data sets

For large data sets some functionality is slightly changed. The correlation is calculated in batches to fit in memory, SVD is skipped because of memory constraint, and for very skewed data, common for large single cell data sets, histograms focusing on parts of the data are added.