o 2i%@sdZddlmZddlmZddlmZddlZddlZddlZddl m Z ddl m Z ddl m Z dd l mZdd l mZdd l mZdd l mZdd l mZddl mZddl mZddl mZddlmZddlmZddZddZGdddejZdS)z"Module for constructing RNN Cells.)absolute_import)division)print_functionN)dtypes)op_def_registry)ops) array_ops)clip_ops)init_ops)math_ops)nn_ops) random_ops) rnn_cell_impl)variable_scope) tf_logging)nestc Cst||||}t|dkr|dS|d}tjd|d}ttjjD] }|j|kr2|Sq't j |d|d}t tjj||S)z4Get a sharded variable concatenated into one tensor.rz/concat/z:0)name) _get_sharded_variablelenvsget_variable_scoperrget_collection GraphKeysCONCATENATED_VARIABLESrconcatadd_to_collection) rshapedtype num_shardssharded_variableZ concat_nameZconcat_full_namevalueZconcat_variabler#E/home/tuu73405/projects/NER-BERT-BiLSTM-CRF--master/models/rnncell.py_get_concat_variables   r%c Cs||dkrtd||ftt|d|}|d||}g}t|D]"}|}||kr3|d7}|tj|d||g|dd|dq'|S)z5Get a list of sharded variables with the given dtype.rz(Too many shards: shape=%s, num_shards=%drz_%dN)r) ValueErrorintmathfloorrangeappendr get_variable) rrrr Zunit_shard_sizeZremaining_rowsshardsi current_sizer#r#r$r,s  " rc sXeZdZdZddddddddejdf fdd Zed d Zed d Z d dZ Z S)CoupledInputForgetGateLSTMCellaLong short-term memory unit (LSTM) recurrent network cell. The default non-peephole implementation is based on: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf S. Hochreiter and J. Schmidhuber. "Long Short-Term Memory". Neural Computation, 9(8):1735-1780, 1997. The peephole implementation is based on: https://research.google.com/pubs/archive/43905.pdf Hasim Sak, Andrew Senior, and Francoise Beaufays. "Long short-term memory recurrent neural network architectures for large scale acoustic modeling." INTERSPEECH, 2014. The coupling of input and forget gate is based on: http://arxiv.org/pdf/1503.04069.pdf Greff et al. "LSTM: A Search Space Odyssey" The class uses optional peep-hole connections, and an optional projection layer. FNrg?Tc stt|j| d| std|||_||_||_||_||_ ||_ ||_ ||_ | |_ | |_| |_|rF| r 0` and `proj_clip` is provided, then the projected values are clipped elementwise to within `[-proj_clip, proj_clip]`. num_unit_shards: How to split the weight matrix. If >1, the weight matrix is stored across num_unit_shards. num_proj_shards: How to split the projection matrix. If >1, the projection matrix is stored across num_proj_shards. forget_bias: Biases of the forget gate are initialized by default to 1 in order to reduce the scale of forgetting at the beginning of the training. state_is_tuple: If True, accepted and returned states are 2-tuples of the `c_state` and `m_state`. By default (False), they are concatenated along the column axis. This default behavior will soon be deprecated. activation: Activation function of the inner states. reuse: (optional) Python boolean describing whether to reuse variables in an existing scope. If not `True`, and the existing scope already has the given variables, an error is raised. )_reusez_%s: Using a concatenated state is slower and will soon be deprecated. Use state_is_tuple=True.N)superr0__init__loggingwarn _num_units_use_peepholes _initializer _num_proj _proj_clip_num_unit_shards_num_proj_shards _forget_bias_state_is_tuple _activationr1rLSTMStateTuple _state_size _output_size) self num_units use_peepholes initializernum_proj proj_clipnum_unit_shardsnum_proj_shards forget_biasstate_is_tuple activationreuse __class__r#r$r4Zs8   z'CoupledInputForgetGateLSTMCell.__init__cC|jSN)rBrDr#r#r$ state_sizez)CoupledInputForgetGateLSTMCell.state_sizecCrRrS)rCrTr#r#r$ output_sizerVz*CoupledInputForgetGateLSTMCell.output_sizec Cs^tj}|jdur |jn|j}|jr|\}}nt|ddgd|jg}t|d|jgd|g}|j}| dd}|j durCt dt d|j |jg|_t d|j|jg|_t d |j|jg|_t d |j |jg|_t d |j|jg|_t d |j|jg|_t d |j |jg|_t d|j|jg|_t j d|jgtd|_t j d|jgtd|_t j d|jgtd|_|t||jt||jt||j|j} d| || |t||jt||j|j} |t||jt||jt| |j|j} | || } |jr#t| | nt| | gd} | | fS)aRun one step of LSTM. Args: inputs: input Tensor, 2D, batch x num_units. state: if `state_is_tuple` is False, this must be a state Tensor, `2-D, batch x state_size`. If `state_is_tuple` is True, this must be a tuple of state Tensors, both `2-D`, with column sizes `c_state` and `m_state`. scope: VariableScope for the created subgraph; defaults to "LSTMCell". Returns: A tuple containing: - A `2-D, [batch x output_dim]`, Tensor representing the output of the LSTM after reading `inputs` when previous state was `state`. Here output_dim is: num_proj if num_proj was set, num_units otherwise. - Tensor(s) representing the new state of LSTM after reading `inputs` when the previous state was `state`. Same type and shape(s) as `state`. Raises: ValueError: If input size cannot be inferred from inputs via static shape inference. Nrr2rz6Could not infer input size from inputs.get_shape()[-1]Z_w_xiZ_w_hiZ_w_ciZ_w_xoZ_w_hoZ_w_coZ_w_xcZ_w_hcZ_b_i)rGZ_b_cZ_b_o) r sigmoidr:r7r?rslicer get_shape with_rankr"r&tfr,Zw_xiZw_hiZw_ciZw_xoZw_hoZw_coZw_xcZw_hcr zeros_initializerb_ib_cb_omatmulr@rrAr)rDinputsstaterYrHc_prevm_prevr input_sizeZi_tc_tZo_tZh_t new_stater#r#r$callsZ        z#CoupledInputForgetGateLSTMCell.call) __name__ __module__ __qualname____doc__r tanhr4propertyrUrWrj __classcell__r#r#rPr$r0>s:  r0) rn __future__rrr collectionsr( tensorflowr]tensorflow.python.frameworkrrrtensorflow.python.opsrr r r r r rrrtensorflow.python.platformrr5Ztensorflow.python.utilrr%rRNNCellr0r#r#r#r$s.