enchanter.addons

Activation functions

Mish

class enchanter.addons.Mish

Bases: torch.nn.modules.module.Module

Apply mish activate function.

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Methods

forward(inputs)

Apply Mish to the input.

forward(inputs: torch.Tensor) → torch.Tensor

Apply Mish to the input.

Examples

>>> import torch
>>> act = Mish()
>>> x = torch.randn(2)
>>> y = act(y)

Parameters

inputs (torch.Tensor) –

Returns

Result of applying mish (torch.Tensor)

Swish

class enchanter.addons.Swish(beta: bool = False)

Bases: torch.nn.modules.module.Module

Apply Swish activate function.

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Methods

forward(x)

Apply Swish function.

forward(x: torch.Tensor) → torch.Tensor

Apply Swish function.

Examples

>>> import torch
>>> act = Swish()
>>> x = torch.randn(2)
>>> y = act(x)

Parameters

x (torch.Tensor) –

Returns

Result of applying Swish (torch.Tensor)

FReLU1d

class enchanter.addons.FReLU1d(in_features: int, kernel_size: int = 3, stride: int = 1, padding: int = 1)

Bases: torch.nn.modules.module.Module

Applies the Funnel Activation (FReLU) for 1d inputs such as sensor signals.

Examples

>>> inputs = torch.randn(1, 3, 128)     # [N, features, seq_len]
>>> frelu = FReLU1d(3)
>>> outputs = frelu(inputs)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Methods

forward(x)

Applies the Funnel Activation (FReLU) for 1d inputs such as sensor signals.

forward(x: torch.Tensor) → torch.Tensor

Applies the Funnel Activation (FReLU) for 1d inputs such as sensor signals.

Parameters

x – torch.Tensor

Returns:

FReLU2d

class enchanter.addons.FReLU2d(in_features: int, kernel_size: int = 3, stride: int = 1, padding: int = 1)

Bases: torch.nn.modules.module.Module

Applies the Funnel Activation (FReLU) for 2d inputs such as images.

Examples

>>> inputs = torch.randn(1, 3, 128, 128)     # [N, channels, heights, widths]
>>> frelu = FReLU2d(3)
>>> outputs = frelu(inputs)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Methods

forward(x)

Applies the Funnel Activation (FReLU) for 2d inputs such as images.

forward(x: torch.Tensor) → torch.Tensor

Applies the Funnel Activation (FReLU) for 2d inputs such as images.

Parameters

x – torch.Tensor

Returns:

Optimizers

TransformerOptimizer

class enchanter.addons.TransformerOptimizer(optimizer: torch.optim.optimizer.Optimizer, d_model: int, warm_up: int)

Bases: object

Reference:

jadore801120/attention-is-all-you-need-pytorch

get_lr() → float

state_dict() → Dict[str, Any]

step() → None

zero_grad() → None