3 GY^$@sdZddlmZddlmZddlmZddlZddlZddlZddl m Z ddl m Z ddl mZdd l mZdd l mZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddZ ddZ!Gdddej"Z#dS)z"Module for constructing RNN Cells.)absolute_import)division)print_functionN)jit)layers)dtypes)op_def_registry)ops) array_ops)clip_ops)init_ops)math_ops)nn_ops) random_ops) rnn_cell_impl)variable_scope) tf_logging)nestc Cst||||}t|dkr"|dS|d}tjjd|d}x$tjtjjD]}|j|krP|SqPWt j |d|d}tj tjj||S)z4Get a sharded variable concatenated into one tensor.rz/concat/z:0)name) _get_sharded_variablelenvsget_variable_scoperr get_collection GraphKeysCONCATENATED_VARIABLESr concatadd_to_collection) rshapedtype num_shardssharded_variable concat_nameconcat_full_namevalueconcat_variabler(8D:\pycharm\NER-BERT-BiLSTM-CRF--master\models\rnncell.py_get_concat_variables   r*c Cs||dkrtd||fttj|d|}|d||}g}xPt|D]D}|}||krh|d7}|jtj|d||g|dd|dqPW|S)z5Get a list of sharded variables with the given dtype.rz(Too many shards: shape=%s, num_shards=%drz_%dN)r!) ValueErrorintmathfloorrangeappendr get_variable) rr r!r"unit_shard_sizeremaining_rowsshardsi current_sizer(r(r)r,s  "rc sXeZdZdZddddddddejdf fdd Zed d Zed d Z d dZ Z S)CoupledInputForgetGateLSTMCellaLong short-term memory unit (LSTM) recurrent network cell. The default non-peephole implementation is based on: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf S. Hochreiter and J. Schmidhuber. "Long Short-Term Memory". Neural Computation, 9(8):1735-1780, 1997. The peephole implementation is based on: https://research.google.com/pubs/archive/43905.pdf Hasim Sak, Andrew Senior, and Francoise Beaufays. "Long short-term memory recurrent neural network architectures for large scale acoustic modeling." INTERSPEECH, 2014. The coupling of input and forget gate is based on: http://arxiv.org/pdf/1503.04069.pdf Greff et al. "LSTM: A Search Space Odyssey" The class uses optional peep-hole connections, and an optional projection layer. FNrg?Tc stt|j| d| s"tjd|||_||_||_||_||_ ||_ ||_ ||_ | |_ | |_| |_|r| rxtj||n|||_||_n | rtj||nd||_||_dS)aInitialize the parameters for an LSTM cell. Args: num_units: int, The number of units in the LSTM cell use_peepholes: bool, set True to enable diagonal/peephole connections. initializer: (optional) The initializer to use for the weight and projection matrices. num_proj: (optional) int, The output dimensionality for the projection matrices. If None, no projection is performed. proj_clip: (optional) A float value. If `num_proj > 0` and `proj_clip` is provided, then the projected values are clipped elementwise to within `[-proj_clip, proj_clip]`. num_unit_shards: How to split the weight matrix. If >1, the weight matrix is stored across num_unit_shards. num_proj_shards: How to split the projection matrix. If >1, the projection matrix is stored across num_proj_shards. forget_bias: Biases of the forget gate are initialized by default to 1 in order to reduce the scale of forgetting at the beginning of the training. state_is_tuple: If True, accepted and returned states are 2-tuples of the `c_state` and `m_state`. By default (False), they are concatenated along the column axis. This default behavior will soon be deprecated. activation: Activation function of the inner states. reuse: (optional) Python boolean describing whether to reuse variables in an existing scope. If not `True`, and the existing scope already has the given variables, an error is raised. )_reusez_%s: Using a concatenated state is slower and will soon be deprecated. Use state_is_tuple=True.N)superr7__init__loggingwarn _num_units_use_peepholes _initializer _num_proj _proj_clip_num_unit_shards_num_proj_shards _forget_bias_state_is_tuple _activationr8rLSTMStateTuple _state_size _output_size) self num_units use_peepholes initializernum_proj proj_clipnum_unit_shardsnum_proj_shards forget_biasstate_is_tuple activationreuse) __class__r(r)r;Zs* z'CoupledInputForgetGateLSTMCell.__init__cCs|jS)N)rI)rKr(r(r) state_sizesz)CoupledInputForgetGateLSTMCell.state_sizecCs|jS)N)rJ)rKr(r(r) output_sizesz*CoupledInputForgetGateLSTMCell.output_sizecCs^tj}|jdkr|jn|j}|jr,|\}}n0tj|ddgd|jg}tj|d|jgd|g}|j}|jj dd}|j dkrt dt j d|j |jg|_t j d|j|jg|_t j d|j|jg|_t j d |j |jg|_t j d |j|jg|_t j d |j|jg|_t j d |j |jg|_t j d |j|jg|_t j d|jgtjd|_t j d|jgtjd|_t j d|jgtjd|_|tj||jtj||jtj||j|j} d| || |jtj||jtj||j|j} |tj||jtj||jtj| |j|j} | |j| } |jrFtj| | ntj| | gd} | | fS)aRun one step of LSTM. Args: inputs: input Tensor, 2D, batch x num_units. state: if `state_is_tuple` is False, this must be a state Tensor, `2-D, batch x state_size`. If `state_is_tuple` is True, this must be a tuple of state Tensors, both `2-D`, with column sizes `c_state` and `m_state`. scope: VariableScope for the created subgraph; defaults to "LSTMCell". Returns: A tuple containing: - A `2-D, [batch x output_dim]`, Tensor representing the output of the LSTM after reading `inputs` when previous state was `state`. Here output_dim is: num_proj if num_proj was set, num_units otherwise. - Tensor(s) representing the new state of LSTM after reading `inputs` when the previous state was `state`. Same type and shape(s) as `state`. Raises: ValueError: If input size cannot be inferred from inputs via static shape inference. Nrrr9z6Could not infer input size from inputs.get_shape()[-1]Z_w_xiZ_w_hiZ_w_ciZ_w_xoZ_w_hoZ_w_coZ_w_xcZ_w_hcZ_b_i)rNZ_b_cZ_b_orZ) r sigmoidrAr>rFr slicer! get_shape with_rankr&r+tfr1Zw_xiZw_hiZw_ciZw_xoZw_hoZw_coZw_xcZw_hcr zeros_initializerb_ib_cb_omatmulrGrrHr)rKinputsstater[rOc_prevm_prevr! input_sizeZi_tc_tZo_tZh_t new_stater(r(r)callsB  ( (( z#CoupledInputForgetGateLSTMCell.call) __name__ __module__ __qualname____doc__r tanhr;propertyrXrYrl __classcell__r(r()rWr)r7>s6  r7)$rp __future__rrr collectionsr- tensorflowr_tensorflow.contrib.compilerr'tensorflow.contrib.layers.python.layersrtensorflow.python.frameworkrrr tensorflow.python.opsr r r r rrrrrtensorflow.python.platformrr<tensorflow.python.utilrr*rRNNCellr7r(r(r(r)s0