API Reference

FastPIDC

A package for inferring undirected gene regulatory (or other) networks from per-node measurements, using information-theoretic algorithms: MI, CLR, PUC and PIDC (Chan, Stumpf & Babtie 2017). The main entry points are get_nodes/infer_network to build an InferredNetwork from a data file, and write_network_file/write_network_npy to save the result.

Node with metadata

Fields:

• label: unique identifying label
• binned_values: data values discretized into bins
• number_of_bins: no. bins the data were discretized into
• probabilities: probability distribution across the bins

Undirected edge

Fields:

• nodes: the two nodes, in an arbitrary order
• weight: weight indicating confidence of edge existing in the true network

Weights are used to rank the edges, and different algorithms may have a different scale. The relative weights within one inferred network are therefore more meaningful than the absolute weight out of context.

PIDCConfig(; backend=:cuda, discretizer="bayesian_blocks",
             estimator="maximum_likelihood", dump_mi_path=nothing,
             dump_puc_path=nothing, verbose=false)

Runtime configuration for PUC/PIDC network inference.

Fields

• backend::Symbol: computation backend, either :cuda (default, requires using CUDA and a functional GPU) or :cpu.
• discretizer::String: discretization method, mirrors the default used by get_nodes.
• estimator::String: probability estimator, mirrors the default used by get_nodes.
• dump_mi_path::Union{Nothing,String}: if set, the pairwise MI score matrix is written to <stem>_mi.npy (see dump_mi_scores).
• dump_puc_path::Union{Nothing,String}: if set, the pre-context PUC score matrix is written to <stem>_puc.npy (see dump_puc_scores).
• verbose::Bool: print progress information while inferring the network.

Throws an ArgumentError if backend is not :cpu or :cuda.

AbstractNetworkInference

Supertype for the network inference algorithms: MINetworkInference, CLRNetworkInference, PUCNetworkInference and PIDCNetworkInference. Used to select behavior in InferredNetwork via the apply_context and get_puc traits.

Mutual information (MI) network inference: raw pairwise MI as edge weights.

Context Likelihood of Relatedness (CLR): MI weights with per-node background context applied.

Proportional Unique Contribution (PUC): redundancy-corrected MI, without context.

Partial Information Decomposition and Context (PIDC): PUC scores with per-node background context applied.

get_nodes(data_file_path::String; <keyword arguments>) -> Vector{Node}

Gets an array of all Nodes from a data file. Dispatches to get_nodes_h5 for .h5 files, or to the whitespace/delimited text loader otherwise (see the text-file method below for its format and keyword arguments).

get_nodes_h5(data_file_path::String; <keyword arguments>) -> Vector{Node}

Gets an array of all Nodes from an HDF5 (.h5) expression file.

The file must contain a "gene_names" dataset, and an expression matrix under one of "matrix_sparse_csc", "matrix_dense", "X", "matrix" or "data" (checked in that order). The matrix may be a dense HDF5 dataset (assumed to be (genes, cells) in C order, as written by AnnData/scanpy, and transposed on load) or an HDF5 group holding a CSC sparse matrix ("data", "indices", "indptr" datasets plus a "shape" attribute, using Python's 0-based indexing, which is converted to Julia's 1-based indexing on load). In both cases the on-disk layout is (cells, genes) after loading.

Arguments

• data_file_path: path to the .h5 file.
• discretizer="bayesian_blocks": algorithm for discretizing the data.
• estimator="maximum_likelihood": algorithm for estimating probabilities.
• number_of_bins=10: will be overwritten if using "bayesian_blocks".

Throws an ArgumentError if the expected datasets, matrix key or matrix group members/attributes are missing.

get_nodes(data_file_path::String; <keyword arguments>)

Gets an array of all Nodes from a data file. It is assumed that the first line of the file is headers (which are discarded) and the subsequent lines each represent one node, and are of the form:

Label datavalue1 datavalue2 ...

though a different delimiter may be specified.

Arguments:

• data_file_path: path to the data file
• delim=false: the file's delimiter. Leave as false if it is whitespace
• discretizer="bayesian_blocks": algorithm for discretizing the data
• estimator="maximum_likelihood": algorithm for estimating probabilities
• number_of_bins=10: will be overwritten if using "bayesian_blocks"

The "maximum_likelihood" estimator is recommended for PUC and PIDC.

infer_network(data_file_path::String, inference::AbstractNetworkInference; <keyword arguments>)

Infers a network, given a data file and a network inference algorithm. It is assumed that the first line of the file is headers (which are discarded) and the subsequent lines each represent one node, and are of the form:

Label datavalue1 datavalue2 ...

though a different delimiter may be specified.

Arguments:

• data_file_path: path to the data file
• inference: network inference algorithm (e.g. PIDCNetworkInference())
• delim=false: the file's delimiter. Leave as false if it is whitespace
• discretizer="bayesian_blocks": algorithm for discretizing the data
• estimator="maximum_likelihood": algorithm for estimating probabilities
• number_of_bins=10: will be overwritten if using "bayesian_blocks"
• base=2: base for the information measures
• out_file_path="": path to output file. If empty, will not write a file

The "maximum_likelihood" estimator is recommended for PUC and PIDC.

InferredNetwork type. Represents a weighted, fully connected network, where an edges's weight indicates the relative confidence of that edge existing in the true network.

Fields:

• nodes: array of all the nodes, in an arbitrary order
• edges: array of all the edges, in descending order of weight

write_network_file(file_path::String, inferred_network::InferredNetwork)

Writes a network file from an InferredNetwork type. Each line of the file will contain an edge, and since networks are assumed undirected, each edge will be written in both directions with the same weight:

...

LabelX LabelY WeightXY

LabelY LabelX WeightXY

...

Arguments:

• file_path: path to the output file
• inferred_network: network that was inferred

write_network_npy(file_path::String, inferred_network::InferredNetwork)

Writes an inferred undirected weighted network as a dense NumPy binary file (.npy) plus a sidecar gene list file preserving row/column order.

Outputs:

• file_path : N x N dense weighted adjacency matrix in .npy format (Float32)
• <stem>_genes.txt : one gene label per line, matching matrix row/column order

To load in Python: import numpy as np

A = np.load("network.npy")

with open("network_genes.txt") as f:
    genes = [line.strip() for line in f]

read_network_file(file_path::AbstractString)

Reads a network file and creates an InferredNetwork type. Assumes that the input is such that each line contains an edge and each edge is written in both directions with the same weight:

...

LabelX LabelY WeightXY

LabelY LabelX WeightXY

...

get_adjacency_matrix(inferred_network::InferredNetwork, threshold = 1.0; <keyword arguments>)

Gets an adjacency matrix given an InferredNetwork and a threshold.

Arguments:

• inferred_network: network that was inferred
• threshold=0.1: threshold above which to keep edges in the network
• absolute=false: interpret threshold as an absolute confidence score

If absolute is false, threshold will be interpreted as the percentage of edges to keep.

dump_mi_scores(mi_scores, nodes, config)

Write the full symmetric MI score matrix to <stem>_mi.npy as Float64. The matrix uses the same <stem>_genes.txt row/column sidecar as the final network output.

dump_puc_scores(scores, nodes, config)

Write the full symmetric pre-context PUC score matrix to <stem>_puc.npy as Float64. The matrix uses the same <stem>_genes.txt row/column sidecar as the final network output.

apply_context(inference::AbstractNetworkInference) -> Bool

Whether inference applies background-context weighting (via get_weights) to the raw scores. true for CLRNetworkInference and PIDCNetworkInference.

get_puc(inference::AbstractNetworkInference) -> Bool

Whether inference computes PUC (redundancy-corrected) scores rather than raw MI. true for PUCNetworkInference and PIDCNetworkInference.

get_weight(edge::Edge) -> Float64

Accessor returning edge.weight, used as the sort key when ordering edges by confidence.

get_joint_probabilities(node1, node2, estimator) -> (probabilities, probabilities1, probabilities2)

Estimate the joint probability distribution probabilities for node1 and node2 (a matrix over their bin ids) using estimator, along with the marginal distributions probabilities1 and probabilities2 recovered by summing over the other node's bins.

get_mi_scores(nodes, number_of_nodes, estimator, base; config=PIDCConfig()) -> SharedMatrix{Float64}

Compute the pairwise mutual information between all nodes, returning a symmetric number_of_nodes by number_of_nodes matrix (the diagonal is left as zero). estimator selects the probability estimator and base the logarithm base. The computation is distributed across worker processes if any are available; config.verbose enables progress printouts.

get_puc_scores(nodes, number_of_nodes, estimator, base; config=PIDCConfig()) -> (mi_scores, puc_scores)

Compute the pre-context Proportional Unique Contribution (PUC) scores for all pairs of nodes (see compute_puc_full for the algorithm), returning both the pairwise MI matrix mi_scores and the PUC score matrix puc_scores. estimator selects the probability estimator, base the logarithm base, and config selects the computation backend (:cpu or :cuda) and enables progress printouts when config.verbose is set.

get_weights(inference, scores, number_of_nodes, nodes) -> SharedMatrix{Float64}

Apply background-context weighting to raw pairwise scores (MI for CLRNetworkInference, PUC for PIDCNetworkInference), returning a new score matrix of edge weights.

For each node, a background distribution of its scores against all other nodes is used to standardize its scores: for PIDCNetworkInference a Gamma distribution is fit to the background (falling back to a CLR-style z-score if the fit fails for either node in a pair), and for CLRNetworkInference a z-score against the background mean/variance is always used. See Chan, Stumpf & Babtie (2017) for details.

build_sorted_edges(nodes, weights) -> Vector{Edge}

Build an Edge for every pair of nodes, weighted by the corresponding entry of the weights matrix, and return them sorted in descending order of weight.

Cache of information measures for an ordered pair of nodes.

Fields:

• mi: mutual information between the two nodes
• si: specific information of the first node with respect to the second

get_mi_and_si(node1::Node, node2::Node, estimator, base) -> (mi, si1, si2)

Compute the mutual information mi between node1 and node2, along with the specific information of each node with respect to the other (si1 for node1, si2 for node2), using probabilities estimated with estimator and logarithms of base base.

compute_puc_full(nodes::Vector{Node}; estimator="maximum_likelihood",
                  base=2, config=PIDCConfig()) -> (mi_scores, puc_scores)

Compute the full pairwise MI matrix and pre-context Proportional Unique Contribution (PUC) matrix for nodes.

For every ordered triple (x, y, z) of distinct nodes, the redundancy between sources x and y with respect to target z is computed from cached specific-information values, and PUC(x, z) += (MI(x,z) - redundancy) / MI(x,z) (clamped to be non-negative, and skipped when MI(x,z) is ~0) is accumulated symmetrically. This is the "sorting trick" formulation, reducing complexity from O(N^3 B) to O(N^2 B log N) relative to a naive implementation (N = number of nodes, B = number of bins).

Arguments

• nodes: nodes to score.
• estimator="maximum_likelihood": probability estimator.
• base=2: logarithm base for the information measures.
• config=PIDCConfig(): selects the computation backend (config.backend, :cpu or :cuda) and verbosity. When :cuda is requested, dispatches to compute_puc_full_cuda (from the CUDA package extension) and errors if that extension is not loaded.

Returns a tuple (mi_scores, puc_scores), each a dense length(nodes)-by-length(nodes) matrix.

compute_puc_full_cuda(nodes, config, base)

GPU implementation of compute_puc_full, defined by the FastPIDCCUDAExt package extension (loaded automatically when using CUDA and a functional GPU is available). Calling this without the extension loaded raises a MethodError; compute_puc_full checks for the method's existence before dispatching to it.

AbstractDiscretizer{N,D}

Supertype for discretizers that encode values of natural type N into discrete bin ids of type D.

DiscretizationAlgorithm

Supertype for algorithms that compute bin edges for a data array (see binedges).

LinearDiscretizer{N,D} <: AbstractDiscretizer{N,D}

Encodes values into bins defined by a sorted list of edges.

Fields

• binedges::Vector{N}: bin edges, sorted smallest to largest.
• nbins::Int: number of bins, i.e. length(binedges) - 1.
• force_outliers_to_closest::Bool: if true, values outside binedges are assigned to the nearest bin instead of raising a BoundsError.

encode(ld::LinearDiscretizer, x) -> Integer

Return the bin id that value x falls into, according to ld's bin edges. When x falls outside the edges, it is either clamped to the nearest end bin (if ld.force_outliers_to_closest) or a BoundsError is thrown. When x is an AbstractArray, encode is broadcast element-wise and the result reshaped to match x's shape.

DiscretizeUniformWidth(nbins) <: DiscretizationAlgorithm

Discretize data into nbins bins of equal width, spanning the data's range (see binedges).

DiscretizeUniformCount(nbins) <: DiscretizationAlgorithm

Discretize data into nbins bins each containing (as close to) an equal number of data points, by placing edges at the midpoints between sorted data values (see binedges).

DiscretizeBayesianBlocks <: DiscretizationAlgorithm

Adaptive discretization that chooses both the number and placement of bins by maximizing a Bayesian blocks fitness function over the (sorted) data, following Scargle (2012). Unlike DiscretizeUniformWidth and DiscretizeUniformCount, the number of bins is not fixed in advance; it is determined by binedges.

binedges(alg::DiscretizeUniformWidth, data) -> Vector

Compute alg.nbins + 1 bin edges spaced evenly across extrema(data).

binedges(alg::DiscretizeUniformCount, data) -> Vector

Compute alg.nbins + 1 bin edges such that each bin contains an equal (or as close to equal as possible) number of sorted data points. Errors if data has fewer points than alg.nbins, or if any resulting edges coincide (non-unique bin edges).

binedges(alg::DiscretizeBayesianBlocks, data) -> Vector

Compute Bayesian-blocks bin edges for data, following the histogram variant of the algorithm in Scargle (2012) (event data, sorted then binned by maximizing a fitness function via dynamic programming). The number of edges returned, and hence the number of bins, is chosen adaptively.

get_bin_ids!(values_x, mode, number_of_bins, bin_ids) -> Int

Discretize values_x in place into bin_ids (a pre-allocated array of the same length), using discretization method mode.

Arguments

• values_x: array of raw (continuous) data values.
• mode: discretization method — one of "bayesian_blocks", "uniform_width", "uniform_count" or "binarize". Falls back to "uniform_width" (with a printed message) if mode is unrecognized, or if the requested method fails on this data.
• number_of_bins: number of bins to use; ignored (and overwritten in the return value) when mode == "bayesian_blocks", since that method chooses its own bin count.
• bin_ids: pre-allocated output array, filled in place with the bin id of each value in values_x.

Returns the actual number of bins used, which differs from number_of_bins when mode == "bayesian_blocks" or when all values in values_x are equal (in which case a single bin is used).

get_frequencies_from_bin_ids(bin_ids_x, number_of_bins_x) -> Vector{Int}

Count how many values fall into each of number_of_bins_x bins, given bin_ids_x, the bin id assigned to each value.

get_frequencies_from_bin_ids(bin_ids_x, bin_ids_y, number_of_bins_x, number_of_bins_y) -> Matrix{Int}

Count the joint frequency of each (bin_ids_x, bin_ids_y) pair, returning a number_of_bins_x by number_of_bins_y matrix of counts.

get_probabilities(estimator, frequencies; <keyword arguments>)

Estimate probabilities from a set of discrete values.

Arguments:

• estimator: the entropy estimator.
• frequencies: the bin frequencies for the discretized data values.
• lambda=nothing: the shrinkage instensity, only used if estimator is "shrinkage".
• prior=1: the Dirichlet prior, only used if estimator is "dirichlet".

get_probabilities_dirichlet(frequencies, prior) -> Array{Float64}

Estimate probabilities from frequencies using a Dirichlet estimator with concentration prior added to every bin before normalizing.

get_probabilities_maximum_likelihood(frequencies) -> Array{Float64}

Estimate probabilities as the simple relative frequencies frequencies / sum(frequencies).

get_probabilities_shrinkage(frequencies, lambda::Nothing) -> Array{Float64}

Estimate probabilities via James-Stein shrinkage towards the uniform distribution, estimating the shrinkage intensity automatically (via get_lambda) since lambda is nothing.

get_probabilities_shrinkage(frequencies, lambda) -> Array{Float64}

Estimate probabilities via James-Stein shrinkage towards the uniform distribution, using the fixed shrinkage intensity lambda.

apply_shrinkage_formula(normalized_frequencies, target, lambda)

Blend normalized_frequencies with target by shrinkage intensity lambda: lambda * target + (1 - lambda) * normalized_frequencies.

get_uniform_distribution(frequencies) -> Float64

Probability of a single bin under a uniform distribution over length(frequencies) bins, i.e. 1 / length(frequencies).

get_normalized_frequencies(frequencies) -> Array{Float64}

Relative frequencies frequencies / sum(frequencies).

get_lambda(normalized_frequencies, target, n) -> Float64

Estimate the James-Stein shrinkage intensity given already-normalized frequencies, a target distribution, and sample size n, following Hausser & Strimmer (2009). Returns 1.0 when n is 0 or 1. The result is clamped to [0, 1].

get_lambda(frequencies, get_target=get_uniform_distribution) -> Float64

Estimate the James-Stein shrinkage intensity directly from raw frequencies, computing the target distribution via get_target and normalizing internally. Returns 1 when the total count is 0 or 1.

remove_non_finite(x)

Return x unchanged if finite, otherwise zero(x). Used to silence NaN/Inf contributions (e.g. 0 * log(0) terms) in information-measure sums.

apply_mutual_information_formula(p_xy, p_x, p_y, base) -> Float64

Compute the mutual information sum(p_xy .* log(base, p_xy ./ (p_x .* p_y))) between two variables, given their joint probabilities p_xy and marginal probabilities p_x, p_y, using logarithms of base base. Non-finite terms (arising from zero probabilities) are treated as zero.

apply_specific_information_formula(p_xz, p_x, p_z, dim_sum, base) -> Vector{Float64}

Compute the specific information of a source variable with respect to a target variable, given their joint probabilities p_xz and marginals p_x (source) and p_z (target). Summation is performed along dimension dim_sum of p_xz (the source's axis), so the result has one value per target bin. Logarithms use base base; non-finite terms are treated as zero.

apply_redundancy_formula(p_z, specific_information_1, specific_information_2, base) -> Float64

Compute the redundancy between two source variables with respect to a common target, as the expectation (over the target's marginal distribution p_z) of the minimum of their specific informations specific_information_1 and specific_information_2. base is accepted for a consistent call signature with the other information-measure formulae but does not affect the computation (the specific informations already encode the logarithm base they were computed with).

_npy_output_path(file_path) -> String

Replace the extension of file_path with .npy, preserving the stem.

_score_output_path(file_path, score_name) -> String

Build the .npy output path for a score dump named score_name (:mi or :puc), appending a _mi/_puc suffix to the stem of file_path unless it is already present. Throws an ArgumentError for any other score_name.

_network_genes_path(file_path) -> String

Path of the gene-label sidecar file (<stem>_genes.txt) that accompanies an inferred-network .npy dump at file_path.

_score_genes_path(file_path, score_name) -> String

Path of the gene-label sidecar file (<stem>_genes.txt) that accompanies the score_name (:mi or :puc) .npy dump derived from file_path.

_write_genes_file(file_path, nodes)

Write one Node label per line to file_path, in the order given by nodes, to serve as the row/column key for a companion .npy matrix.

FastPIDCCUDAExt

Package extension providing a CUDA-accelerated implementation of FastPIDC.compute_puc_full_cuda, loaded automatically once using CUDA makes the CUDA package available alongside FastPIDC. Selected by passing config.backend = :cuda (the default) to FastPIDC.compute_puc_full.

joint_counts_kernel_chunked!(data, counts, n, m, k_bins, z_start, z_chunk_size)

CUDA kernel: for a chunk of z_chunk_size target genes starting at z_start, accumulate the joint bin-count histogram counts[u, v, x, z_local] (co-occurrences of bin u for gene x and bin v for gene z_global = z_start + z_local - 1) across all m samples in data. One GPU thread handles one (x, z_local) pair; n is the number of genes and k_bins the number of discretization bins.

mi_si_kernel_chunked!(counts, marginals, mi_matrix, si_matrix, n, m, k_bins, z_start, z_chunk_size)

CUDA kernel: from the joint bin counts counts (as produced by joint_counts_kernel_chunked!) and per-gene marginal bin probabilities marginals, compute the mutual information mi_matrix[x, z_global] and the specific information si_matrix[:, x, z_local] of gene x with respect to target gene z_global = z_start + z_local - 1, for the chunk of z_chunk_size targets starting at z_start. One GPU thread handles one (x, z_local) pair; n is the number of genes, m the number of samples, and k_bins the number of discretization bins.

puc_accumulation_kernel_chunked!(si_matrix, mi_matrix, puc_scores, marginals, n, k_bins, z_start, z_chunk_size)

CUDA kernel: for each target gene z_global in the current chunk and each source gene x, accumulate the PUC contribution puc_scores[x, z_global] += (MI(x, z_global) - redundancy(x, y, z_global)) / MI(x, z_global) (clamped to be non-negative) summed over all other genes y, using specific information values from si_matrix and marginal probabilities from marginals. One GPU thread handles one (x, z_local) pair, looping internally over y; n is the number of genes and k_bins the number of discretization bins. Contributions still need to be symmetrized (puc_scores[i,j] + puc_scores[j,i]) by the caller, since each thread only writes puc_scores[x, z_global].