mindformers.modules¶

MindFormers Transformers API.

mindformers.modules.layers¶

`mindformers.modules.layers.Dropout`	A Dropout Implements with P.Dropout and P.DropoutDoMask for parallel training.
`mindformers.modules.layers.FixedSparseAttention`	Fixed Sparse Attention Layer.
`mindformers.modules.layers.LayerNorm`	A self-defined layer norm operation using reduce sum and reduce mean
`mindformers.modules.layers.Linear`	The dense connected layer.

mindformers.modules.transformer¶

`mindformers.modules.transformer.AttentionMask`	Get the Lower triangular matrix from the input mask.
`mindformers.modules.transformer.EmbeddingOpParallelConfig`	The parallel config of `VocabEmbedding` for the setting data parallel or model parallel for the embedding table.
`mindformers.modules.transformer.FeedForward`	The multilayer perceptron with two linear layers with dropout applied at final output.
`mindformers.modules.transformer.MoEConfig`	The configuration of MoE (Mixture of Expert).
`mindformers.modules.transformer.MultiHeadAttention`	This is an implementation of multihead attention in the paper Attention is all you need.
`mindformers.modules.transformer.OpParallelConfig`	OpParallelConfig for the setting data parallel and model parallel.
`mindformers.modules.transformer.Transformer`	Transformer module including encoder and decoder.
`mindformers.modules.transformer.TransformerDecoder`	Transformer Decoder module with multi-layer stacked of TransformerDecoderLayer, including multihead self attention, cross attention and feedforward layer.
`mindformers.modules.transformer.TransformerDecoderLayer`	Transformer Decoder Layer.
`mindformers.modules.transformer.TransformerEncoder`	Transformer Encoder module with multi-layer stacked of TransformerEncoderLayer, including multihead self attention and feedforward layer.
`mindformers.modules.transformer.TransformerEncoderLayer`	Transformer Encoder Layer.
`mindformers.modules.transformer.TransformerOpParallelConfig`	TransformerOpParallelConfig for setting parallel configuration, such as the data parallel and model parallel.
`mindformers.modules.transformer.TransformerRecomputeConfig`	TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers.
`mindformers.modules.transformer.VocabEmbedding`	The embedding lookup table from the 0-th dim of the parameter table.