transformer model based on Tensorlayer by ArnoldLIULJ · Pull Request #1027 · tensorlayer/TensorLayer

ArnoldLIULJ · 2019-07-22T09:23:06Z

Checklist

I've tested that my changes are compatible with the latest version of Tensorflow.
I've read the Contribution Guidelines
I've updated the documentation if necessary.

Motivation and Context

Description

ArnoldLIULJ · 2019-07-22T09:25:23Z

Documentations haven't been done

zsdonghao · 2019-07-22T09:30:29Z

tensorlayer/models/transformer/attention_layer.py

+import tensorlayer as tl
+
+
+class MultiHeadAttentionLayer(tl.layers.Layer):


MultiHeadAttention is better than MultiHeadAttentionLayer?

zsdonghao · 2019-07-22T09:31:09Z

tensorlayer/models/transformer/utils/optimizer.py

+K = tf.keras.backend
+
+
+class LazyAdam(tf.keras.optimizers.Adam):


move this part to tl.optimizer ??

OKAY.
Also, should I modify the "copyright" in the top? I reference most of the codes from TensorFlow officials.

…nto transformer

JingqingZ · 2019-07-22T10:02:45Z

The dependency on Keras should be removed gradually.

tf.keras.optimizers -> tf.optimiziers
tf.keras.initializers -> tl.initializers

zsdonghao · 2019-09-01T02:01:51Z

ready to merge?

ArnoldLIULJ · 2019-09-01T10:16:49Z

Add attention visualisation util
Add attention visualisation test

ArnoldLIULJ · 2019-09-01T10:17:03Z

ready to merge?

not yet

ArnoldLIULJ · 2019-09-02T09:52:24Z

Add documentation
Add attention-weights visualisation and pass unit-testing
READY TO MERGE

zsdonghao · 2019-09-02T13:30:41Z

Hi, could you provide an example code in the examples folder? and update changelog.md ? thanks

zsdonghao · 2019-09-02T13:40:28Z

tensorlayer/models/transformer/attention_layer.py

+    """The :class:`MultiHeadAttentionLayer` layer is for multi-head attention computation.
+    The weight computation is between "key" and "query", which will then matmul with "value" to generate information
+    that selectively focuses on the "query" messages.
+    Parameters


missing space

zsdonghao · 2019-09-02T13:41:00Z

tensorlayer/models/transformer/beamsearchHelper/beam_search.py

+):
+    """Search for sequence of subtoken ids with the largest probability.
+
+    Args:


incorrect RST format, should be Parameters with underline

luomai · 2019-09-02T14:17:28Z

There are some naming issues with this PR. Please don't merge it for now.

luomai · 2019-09-02T14:18:45Z

tensorlayer/optimizers/lazyAdam.py

+K = tf.keras.backend
+
+
+class LazyAdam(tf.optimizers.Adam):


LazyAdamOptimizer

luomai · 2019-09-02T14:19:02Z

tensorlayer/optimizers/lazyAdam.py

@@ -0,0 +1,147 @@
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.


Rename the file to lazy_adam.py

luomai · 2019-09-02T14:19:34Z

tensorlayer/optimizers/lazyAdam.py

+        self.verbose = verbose
+        if init_steps is None:
+            init_steps = 0.0
+        self.steps = float(init_steps)  # Total steps during training.


Why steps is a float not an int?

I will get rid of this part. It has been updated in tf.keras library. thanks

luomai · 2019-09-02T14:20:35Z

tensorlayer/models/transformer/feedforward_layer.py

+import tensorlayer as tl
+
+
+class FeedForwardLayer(tl.layers.Layer):


How is this FeedForwardLayer different from conventioinal feedforward layer? If this is a special implementation, can it have a more specific name?

FeedforwardLayer here is specifically designed in Transformer model, which includes two transformations.
The conventional feedforward layer only contains one.
Any suggestion on how the naming?

TransformerFeedFordwardLayer?

luomai · 2019-09-02T14:23:38Z

tensorlayer/optimizers/lazyAdam.py

+
+import numpy as np
+import tensorflow as tf
+K = tf.keras.backend


Does this has to be a global variable of tensorlayer?

luomai · 2019-09-02T14:24:50Z

tensorlayer/models/transformer/beamsearchHelper/beam_search.py

+import tensorflow as tf
+import tensorlayer.models.transformer.beamsearchHelper.beam_search_v1 as v1
+
+_StateKeys = v1._StateKeys  # pylint: disable=protected-access


We should avoid global variables in the library as much as possible.

I will try to optimise this one.

luomai · 2019-09-12T10:21:57Z

@ArnoldLIULJ any update?

ArnoldLIULJ · 2019-09-13T10:13:43Z

@ArnoldLIULJ any update?

was on vocation and would be working on a simplified tutorial today

ArnoldLIULJ · 2019-09-13T22:46:55Z

Add examples in example/translation_task/tutorial_transformer

ArnoldLIULJ · 2019-09-13T22:51:08Z

Hi, could you provide an example code in the examples folder? and update changelog.md ? thanks

done

zsdonghao · 2019-09-14T08:54:06Z

Hi the RST format is not correct in many function, please check~

ArnoldLIULJ · 2019-09-18T13:03:37Z

please check

zsdonghao · 2019-09-23T02:13:42Z

I think this one can be merged after the travis pass~.

transformer updated

5b85af7

zsdonghao reviewed Jul 22, 2019

View reviewed changes

zsdonghao and others added 4 commits July 22, 2019 17:31

Merge branch 'master' into transformer

4d19a5a

minor change

d1a20df

Merge branch 'master' of https://github.com/tensorlayer/tensorlayer i…

2e44277

…nto transformer

minor

7717b74

zsdonghao and others added 9 commits July 26, 2019 16:57

Merge branch 'master' into transformer

1b349bb

merge

1df422c

adjust files

21161cb

Merge branch 'master' into transformer

412eadf

merge

6ecca88

attention visualisation

005ab91

add attention visualisation

8911654

optimizer update

61bf27f

fix

3ef8d8b

Lingjun Liu added 2 commits September 1, 2019 11:15

add attention visualisation

048d9a3

add attention visualisation

a47aee1

Lingjun Liu added 4 commits September 1, 2019 12:24

add decoder part attention visualisation

3c4cae1

documentation

4d2e19e

documentation

f5438a7

documentation

a48e1d3

zsdonghao reviewed Sep 2, 2019

View reviewed changes

luomai requested changes Sep 2, 2019

View reviewed changes

Lingjun Liu added 2 commits September 13, 2019 23:07

add examples

90d536e

documentation

e2662c2

documentation

e0e81f0

Lingjun Liu added 2 commits September 14, 2019 11:18

doc

80c985c

doc

2f316b0

ArnoldLIULJ force-pushed the transformer branch from 7b3be01 to 2f316b0 Compare September 14, 2019 10:27

Lingjun Liu and others added 4 commits September 14, 2019 11:28

reverse change

990e014

reverse change

2c1ced8

doc

9144165

optimizer

576af52

Lingjun Liu and others added 2 commits September 18, 2019 14:06

doc

a2a1cbf

Merge branch 'master' into transformer

0670f1c

		import tensorlayer as tl


		class MultiHeadAttentionLayer(tl.layers.Layer):

		K = tf.keras.backend


		class LazyAdam(tf.keras.optimizers.Adam):

		@@ -0,0 +1,147 @@
		# Copyright 2019 The TensorFlow Authors. All Rights Reserved.

		import tensorlayer as tl


		class FeedForwardLayer(tl.layers.Layer):

Conversation

ArnoldLIULJ commented Jul 22, 2019

Checklist

Motivation and Context

Description

Uh oh!

ArnoldLIULJ commented Jul 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingqingZ commented Jul 22, 2019

Uh oh!

zsdonghao commented Sep 1, 2019

Uh oh!

ArnoldLIULJ commented Sep 1, 2019

Uh oh!

ArnoldLIULJ commented Sep 1, 2019

Uh oh!

ArnoldLIULJ commented Sep 2, 2019

Uh oh!

zsdonghao commented Sep 2, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luomai commented Sep 2, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luomai commented Sep 12, 2019

Uh oh!

ArnoldLIULJ commented Sep 13, 2019

Uh oh!

ArnoldLIULJ commented Sep 13, 2019

Uh oh!

ArnoldLIULJ commented Sep 13, 2019

Uh oh!

zsdonghao commented Sep 14, 2019

Uh oh!

ArnoldLIULJ commented Sep 18, 2019

Uh oh!

zsdonghao commented Sep 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants