cross entropy loss_tf.nn.sampled_softmax_loss 示例

文法結構：

tf.nn.sampled_softmax_loss(
    weights, biases, labels, inputs, num_sampled, num_classes, num_true=1,
    sampled_values=None, remove_accidental_hits=True, seed=None,
    name='sampled_softmax_loss'
)
           

weights

A

Tensor

of shape

[num_classes,dim]

, or a list of

Tensor

objects whose concatenation along dimension 0 has shape [num_classes, dim]. The (possibly-sharded) class embeddings.

biases

A

Tensor

of shape

[num_classes]

. The class biases.

labels

A

Tensor

of type

int64

and shape

[batch_size,num_true]

. The target classes. Note that this format differs from the

labels

argument of

nn.softmax_cross_entropy_with_logits

.

inputs

A

Tensor

of shape

[batch_size,dim]

. The forward activations of the input network.

num_sampled

An

int

. The number of classes to randomly sample per batch.

num_classes

An

int

. The number of possible classes.

num_true

An

int

. The number of target classes per training example.

sampled_values

a tuple of (

sampled_candidates

,

true_expected_count

,

sampled_expected_count

) returned by a

*_candidate_sampler

function. (if None, we default to

log_uniform_candidate_sampler

)

remove_accidental_hits

A

bool

. whether to remove "accidental hits" where a sampled class equals one of the target classes. Default is True.

seed

random seed for candidate sampling. Default to None, which doesn't set the op-level random seed for candidate sampling.

name

A name for the operation (optional).

示例：

import tensorflow as tf

a = tf.nn.sampled_softmax_loss(weights=tf.constant([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]),
                               # [num_classes, dim] = [10, 2]
                               biases=tf.constant([0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]),
                               # [num_classes] = [10]
                               labels=tf.constant([[2], [3], [5]]),
                               # [batch_size, num_true] = [3, 1]
                               inputs=tf.constant([[0.2, 0.1], [0.4, 0.1], [0.22, 0.12]]),
                               # [batch_size, dim] = [3, 2]
                               
                               num_sampled=3,
                               num_classes=10,
                               num_true = 1,
                               seed = 2020,
                               name = "sampled_softmax_loss"
                          )
           

結果：

with tf.compat.v1.Session() as sess:
    print(sess.run(a))

# [0.37478584 0.2859666  0.0703702 ]
           

在源碼的介紹中，train的方式和eval的方式是分開的：

if mode == "train":
    loss = tf.nn.sampled_softmax_loss(
        weights=weights,
        biases=biases,
        labels=labels,
        inputs=inputs,
        ...)

elif mode == "eval":
    logits = tf.matmul(inputs, tf.transpose(weights))
    logits = tf.nn.bias_add(logits, biases)
    labels_one_hot = tf.one_hot(labels, n_classes)
    loss = tf.nn.softmax_cross_entropy_with_logits(
        labels=labels_one_hot,
        logits=logits)
           

train步驟中除了選擇一批label作為負樣本外，實作方式與eval中描述的一緻。

參考：

https://tensorflow.google.cn/api_docs/python/tf/nn/sampled_softmax_loss?hl=zh-cn

https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/python/ops/nn_impl.py#L2066-L2154