o 3id @sdZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl ZddlmmZeGdddeZGdd d eZd d Zd d ZddZddZd:ddZd:ddZd:ddZd;ddZ    dd+d,Z dd-d.d.d/ed$d$ddf d0d1Z!d?d2d3Z"d4d5Z#d6d7Z$d:d8d9Z%dS)@z*The main BERT model and related functions.)absolute_import)division)print_functionNc@sVeZdZdZ          dd d Zed d ZeddZddZddZ dS) BertConfigzConfiguration for `BertModel`. gelu皙?{Gz?c CsF||_||_||_||_||_||_||_||_| |_| |_ | |_ dS)aConstructs BertConfig. Args: vocab_size: Vocabulary size of `inputs_ids` in `BertModel`. hidden_size: Size of the encoder layers and the pooler layer. num_hidden_layers: Number of hidden layers in the Transformer encoder. num_attention_heads: Number of attention heads for each attention layer in the Transformer encoder. intermediate_size: The size of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act: The non-linear activation function (function or string) in the encoder and pooler. hidden_dropout_prob: The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob: The dropout ratio for the attention probabilities. max_position_embeddings: The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). type_vocab_size: The vocabulary size of the `token_type_ids` passed into `BertModel`. initializer_range: The stdev of the truncated_normal_initializer for initializing all weight matrices. N) vocab_size hidden_sizenum_hidden_layersnum_attention_heads hidden_actintermediate_sizehidden_dropout_probattention_probs_dropout_probmax_position_embeddingstype_vocab_sizeinitializer_range) selfrrrrrrrrrrrrD/home/tuu73405/projects/NER-BERT-BiLSTM-CRF--master/bert/modeling.py__init__%s$ zBertConfig.__init__cCs,tdd}t|D] \}}||j|<q |S)zAConstructs a `BertConfig` from a Python dictionary of parameters.N)r)rsix iteritems__dict__)clsZ json_objectconfigkeyvaluerrr from_dictUs  zBertConfig.from_dictcCsHtjj|d }|}Wdn1swY|t|S)z9Constructs a `BertConfig` from a json file of parameters.rN)tfiogfileGFilereadr$jsonloads)r json_filereadertextrrrfrom_json_file]s zBertConfig.from_json_filecCst|j}|S)z0Serializes this instance to a Python dictionary.)copydeepcopyr)routputrrrto_dictds zBertConfig.to_dictcCstj|ddddS)z*Serializes this instance to a JSON string.T)indent sort_keys )r+dumpsr4rrrrto_json_stringiszBertConfig.to_json_stringN) rrrrr r r r r r ) __name__ __module__ __qualname____doc__r classmethodr$r0r4r;rrrrr"s& 0   rc@sJeZdZdZ    dddZddZdd Zd d Zd d ZddZ dS) BertModelaBERT model ("Bidirectional Encoder Representations from Transformers"). Example usage: ```python # Already been converted into WordPiece token ids input_ids = tf.constant([[31, 51, 99], [15, 5, 0]]) input_mask = tf.constant([[1, 1, 1], [1, 1, 0]]) token_type_ids = tf.constant([[0, 0, 1], [0, 2, 0]]) config = modeling.BertConfig(vocab_size=32000, hidden_size=512, num_hidden_layers=8, num_attention_heads=6, intermediate_size=1024) model = modeling.BertModel(config=config, is_training=True, input_ids=input_ids, input_mask=input_mask, token_type_ids=token_type_ids) label_embeddings = tf.get_variable(...) pooled_output = model.get_pooled_output() logits = tf.matmul(pooled_output, label_embeddings) ... ``` NFc Cst|}|s d|_d|_t|dd}|d} |d} |dur)tj| | gtjd}|dur7tj| | gtjd}tj |dd t d -t ||j |j |j d |d \|_|_t|jd ||jdd d|j |j|jd |_Wdn1stwYt d(t||} t|j| |j |j|j|jt|j|j|j|j d d |_Wdn1swY|jd|_t d-tj|jddddddfdd} tjjj|j tj t!|j d| |_"Wdn1swYWddSWddS1swYdS)aConstructor for BertModel. Args: config: `BertConfig` instance. is_training: bool. true for training model, false for eval model. Controls whether dropout will be applied. input_ids: int32 Tensor of shape [batch_size, seq_length]. input_mask: (optional) int32 Tensor of shape [batch_size, seq_length]. token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length]. use_one_hot_embeddings: (optional) bool. Whether to use one-hot word embeddings or tf.embedding_lookup() for the word embeddings. scope: (optional) variable scope. Defaults to "bert". Raises: ValueError: The config is invalid or one of the input tensor shapes is invalid. r5 expected_rankrNshapedtypebert) default_name embeddingsword_embeddings) input_idsrembedding_sizerword_embedding_nameuse_one_hot_embeddingsTtoken_type_embeddingsposition_embeddings) input_tensoruse_token_typetoken_type_idstoken_type_vocab_sizetoken_type_embedding_nameuse_position_embeddingsposition_embedding_namerr dropout_probencoder) rSattention_maskrrrrintermediate_act_fnrrrdo_return_all_layersZpooleraxisunits activationkernel_initializer)#r1r2rrget_shape_listr&onesint32zerosvariable_scopeembedding_lookuprrrembedding_outputembedding_tableembedding_postprocessorrr%create_attention_mask_from_input_masktransformer_modelrrrget_activationrall_encoder_layerssequence_outputsqueezekeraslayersDensetanhcreate_initializer pooled_output) rr! is_trainingrM input_maskrUrPscope input_shape batch_size seq_lengthr\Zfirst_token_tensorrrrrs         &5$zBertModel.__init__cC|jSN)rzr:rrrget_pooled_outputzBertModel.get_pooled_outputcCr)zGets final hidden layer of encoder. Returns: float Tensor of shape [batch_size, seq_length, hidden_size] corresponding to the final hidden of the transformer encoder. )rsr:rrrget_sequence_outputszBertModel.get_sequence_outputcCrr)rrr:rrrget_all_encoder_layersrz BertModel.get_all_encoder_layerscCr)aGets output of the embedding lookup (i.e., input to the transformer). Returns: float Tensor of shape [batch_size, seq_length, hidden_size] corresponding to the output of the embedding layer, after summing the word embeddings with the positional embeddings and the token type embeddings, then performing layer normalization. This is the input to the transformer. )rlr:rrrget_embedding_outputs zBertModel.get_embedding_outputcCrr)rmr:rrrget_embedding_table rzBertModel.get_embedding_table)NNFN) r<r=r>r?rrrrrrrrrrrAns k  rAc Cs:ddttdtj|dt|d}||S)zGaussian Error Linear Unit. This is a smoother version of the RELU. Original paper: https://arxiv.org/abs/1606.08415 Args: x: float Tensor to perform activation. Returns: `x` with the GELU activation applied. g??r5gHm?)r&rxnpsqrtpipow)xcdfrrrr s "r cCsbt|tjs|S|s dS|}|dkrdS|dkrtjjS|dkr$tS|dkr+tjSt d|)aMaps a string to a Python function, e.g., "relu" => `tf.nn.relu`. Args: activation_string: String name of the activation function. Returns: A Python function corresponding to the activation function. If `activation_string` is None, empty, or "linear", this will return None. If `activation_string` is not a string, it will return `activation_string`. Raises: ValueError: The `activation_string` does not correspond to a known activation. Nlinearrelur rxzUnsupported activation: %s) isinstancer string_typeslowerr&nnrr rx ValueError)Zactivation_stringZactrrrrqs  rqc Csi}i}t}|D]}|j}td|}|dur|d}|||<q tj|}t}|D]} | d| d}}||vr?q/|||<d||<d||d<q/||fS)zDCompute the union of the current variables and checkpoint variables.z ^(.*):\d+$NrErz:0) collections OrderedDictnamerematchgroupr&trainlist_variables) tvarsinit_checkpointassignment_mapinitialized_variable_namesZname_to_variablevarrm init_varsrrrr"get_assignment_map_from_checkpointDs&    rcCs*|dus|dkr |Stj|d|}|S)aPerform dropout. Args: input_tensor: float Tensor. dropout_prob: Python float. The probability of dropping out a value (NOT of *keeping* a dimension as in `tf.nn.dropout`). Returns: A version of `input_tensor` with dropout applied. NrBr)r&rdropout)rSrZr3rrrr_s rcCstjjj|dd|dS)Creates a `truncated_normal_initializer` with the given range.)stddev)r&truncated_normal_initializer)rrrrrys ryrLFc Cs|jjdkrtj|dgd}tj|||gt|d}t|dg}|r1tj||d}t||} nt ||} t |} t| | dd| d|g} | |fS)a=Looks up words embeddings for id tensor. Args: input_ids: int32 Tensor of shape [batch_size, seq_length] containing word ids. vocab_size: int. Size of the embedding vocabulary. embedding_size: int. Width of the word embeddings. initializer_range: float. Embedding initialization range. word_embedding_name: string. Name of the embedding table. use_one_hot_embeddings: bool. If True, use one-hot method for word embeddings. If False, use `tf.gather()`. Returns: float Tensor of shape [batch_size, seq_length, embedding_size]. r5r_r`rrG initializerdepthr) rGndimsr& expand_dims get_variableryreshapeone_hotmatmulgatherrf) rMrrNrrOrPrmflat_input_idsZone_hot_input_idsr3r~rrrrks"  rkr rQTrRr r c Cs^t|dd} | d} | d} | d} |}|rK|durtdtj||| gt|d}t|d g}tj||d }t||}t|| | | g}||7}|rt| |}t |gGtj||| gt|d}t |ddg| d g}t |j }g}t|dD]}|dq|| | gt||}||7}Wdn1swYt|| }|S) aPerforms various post-processing on a word embedding tensor. Args: input_tensor: float Tensor of shape [batch_size, seq_length, embedding_size]. use_token_type: bool. Whether to add embeddings for `token_type_ids`. token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length]. Must be specified if `use_token_type` is True. token_type_vocab_size: int. The vocabulary size of `token_type_ids`. token_type_embedding_name: string. The name of the embedding table variable for token type ids. use_position_embeddings: bool. Whether to add position embeddings for the position of each token in the sequence. position_embedding_name: string. The name of the embedding table variable for positional embeddings. initializer_range: float. Range of the weight initialization. max_position_embeddings: int. Maximum sequence length that might ever be used with this model. This can be longer than the sequence length of input_tensor, but cannot be shorter. dropout_prob: float. Dropout probability applied to the final output tensor. Returns: float tensor with same shape as `input_tensor`. Raises: ValueError: One of the tensor shapes or input values is invalid. rrCrrEr5Nz>`token_type_ids` must be specified if`use_token_type` is True.rr_r)rfrr&rryrrrassert_less_equalcontrol_dependenciesslicelenrGas_listrangeappendextendr)rSrTrUrVrWrXrYrrrZr~rrwidthr3Ztoken_type_tableZflat_token_type_idsZ one_hot_idsrQ assert_opZfull_position_embeddingsrRnum_dimsZposition_broadcast_shape_rrrrnsV %      rnc Csrt|ddgd}|d}|d}t|dd}|d}tt||d|gtj}tj||dgtjd}||}|S)aCreate 3D attention mask from a 2D tensor mask. Args: from_tensor: 2D or 3D Tensor of shape [batch_size, from_seq_length, ...]. to_mask: int32 Tensor of shape [batch_size, to_seq_length]. Returns: float Tensor of shape [batch_size, from_seq_length, to_seq_length]. r5rrCrrErF)rfr&castrfloat32rg) from_tensorZto_mask from_shaperfrom_seq_lengthto_shape to_seq_lengthZbroadcast_onesmaskrrrros   rorErBcCsdd}t|ddgd}t|ddgd}t|t|kr tdt|dkr3|d} |d} |d} nt|dkrI| d usE| d usE| d urItd t|}t|}tjjj|||d t| d |}tjjj|||d t| d |}tjjj|||dt| d |}||| || |}||| || |}tj ||dd}t |dt t |}|d urtj|dgd}dt|tjd}||7}tj|}t||}t|| | ||g}t|gd}t ||}t|gd}| rt|| | ||g}|St|| | ||g}|S)az Performs multi-headed attention from `from_tensor` to `to_tensor`. This is an implementation of multi-headed attention based on "Attention is all you Need". If `from_tensor` and `to_tensor` are the same, then this is self-attention. Each timestep in `from_tensor` attends to the corresponding sequence in `to_tensor`, and returns a fixed-with vector. This function first projects `from_tensor` into a "query" tensor and `to_tensor` into "key" and "value" tensors. These are (effectively) a list of tensors of length `num_attention_heads`, where each tensor is of shape [batch_size, seq_length, size_per_head]. Then, the query and key tensors are dot-producted and scaled. These are softmaxed to obtain attention probabilities. The value tensors are then interpolated by these probabilities, then concatenated back to a single tensor and returned. In practice, the multi-headed attention are done with transposes and reshapes rather than actual separate tensors. Args: from_tensor: float Tensor of shape [batch_size, from_seq_length, from_width]. to_tensor: float Tensor of shape [batch_size, to_seq_length, to_width]. attention_mask: (optional) int32 Tensor of shape [batch_size, from_seq_length, to_seq_length]. The values should be 1 or 0. The attention scores will effectively be set to -infinity for any positions in the mask that are 0, and will be unchanged for positions that are 1. num_attention_heads: int. Number of attention heads. size_per_head: int. Size of each attention head. query_act: (optional) Activation function for the query transform. key_act: (optional) Activation function for the key transform. value_act: (optional) Activation function for the value transform. attention_probs_dropout_prob: (optional) float. Dropout probability of the attention probabilities. initializer_range: float. Range of the weight initializer. do_return_2d_tensor: bool. If True, the output will be of shape [batch_size * from_seq_length, num_attention_heads * size_per_head]. If False, the output will be of shape [batch_size, from_seq_length, num_attention_heads * size_per_head]. batch_size: (Optional) int. If the input is 2D, this might be the batch size of the 3D version of the `from_tensor` and `to_tensor`. from_seq_length: (Optional) If the input is 2D, this might be the seq length of the 3D version of the `from_tensor`. to_seq_length: (Optional) If the input is 2D, this might be the seq length of the 3D version of the `to_tensor`. Returns: float Tensor of shape [batch_size, from_seq_length, num_attention_heads * size_per_head]. (If `do_return_2d_tensor` is true, this will be of shape [batch_size * from_seq_length, num_attention_heads * size_per_head]). Raises: ValueError: Any of the arguments or tensor shapes are invalid. cSs(t|||||g}t|gd}|S)Nrr5rEr)r&r transpose)rSrrrrrrrrtranspose_for_scoress  z-attention_layer..transpose_for_scoresr5rrCz=The rank of `from_tensor` must match the rank of `to_tensor`.rrENzWhen passing in rank 2 tensors to attention_layer, the values for `batch_size`, `from_seq_length`, and `to_seq_length` must all be specified.query)rcrdrrer"r#T) transpose_brr`gr)rfrrreshape_to_matrixr&rurvrwryrmultiplymathrfloatrrrrsoftmaxrrr)r to_tensorr\r size_per_headZ query_actZkey_actZ value_actrrdo_return_2d_tensorrrrrrrZfrom_tensor_2dZ to_tensor_2dZ query_layerZ key_layerZ value_layerattention_scoresadderZattention_probsZ context_layerrrrattention_layer=sG          rrrrc Cs||dkrtd||ft||} t|dd} | d} | d}| d}||kr2td||ft|}g}t|D]}td||}td g}td t||||| || d | ||d }||Wd n1stwYd }t |dkr|d}ntj |dd}tj j d tj jj|t| d|}t||}t||}Wd n1swYtj j dtj jj||t| d|}Wd n1swYtj j dtj jj|t| d|}Wd n 1swYt||}t||}|}||Wd n 1s#wYWd n 1s3wYq<| rPg}|D] }t|| }||q@|St|| }|S)a.Multi-headed, multi-layer Transformer from "Attention is All You Need". This is almost an exact implementation of the original Transformer encoder. See the original paper: https://arxiv.org/abs/1706.03762 Also see: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py Args: input_tensor: float Tensor of shape [batch_size, seq_length, hidden_size]. attention_mask: (optional) int32 Tensor of shape [batch_size, seq_length, seq_length], with 1 for positions that can be attended to and 0 in positions that should not be. hidden_size: int. Hidden size of the Transformer. num_hidden_layers: int. Number of layers (blocks) in the Transformer. num_attention_heads: int. Number of attention heads in the Transformer. intermediate_size: int. The size of the "intermediate" (a.k.a., feed forward) layer. intermediate_act_fn: function. The non-linear activation function to apply to the output of the intermediate/feed-forward layer. hidden_dropout_prob: float. Dropout probability for the hidden layers. attention_probs_dropout_prob: float. Dropout probability of the attention probabilities. initializer_range: float. Range of the initializer (stddev of truncated normal). do_return_all_layers: Whether to also return all layers or just the final layer. Returns: float Tensor of shape [batch_size, seq_length, hidden_size], the final hidden layer of the Transformer. Raises: ValueError: A Tensor shape or parameter is invalid. rzLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rrCrEr5z6The width of the input tensor (%d) != hidden size (%d)zlayer_%d attentionrT) rrr\rrrrrrrrNr_r`r3)rcre intermediaterb)rintrfrrr&rjrrrconcatcompatv1rurvrwryrrreshape_from_matrix)rSr\rrrrr]rrrr^Zattention_head_sizer~rr input_width prev_outputZall_layer_outputsZ layer_idx layer_inputZattention_headsZattention_headattention_outputZintermediate_output layer_output final_outputs final_outputrrrrps 0            M  rpcCs|dur|j}|durt||||j}g}t|D] \}}|dur)||q|s.|St|}|D]}||||<q5|S)a(Returns a list of the shape of tensor, preferring static dimensions. Args: tensor: A tf.Tensor object to find the shape of. expected_rank: (optional) int. The expected rank of `tensor`. If this is specified and the `tensor` has a different rank, and exception will be thrown. name: Optional name of the tensor for the error message. Returns: A list of dimensions of the shape of tensor. All static dimensions will be returned as python integers, and dynamic dimensions will be returned as tf.Tensor scalars. N)r assert_rankrGr enumeraterr&)tensorrDrrGZnon_static_indexesindexdimZ dyn_shaperrrrfs     rfcCsH|jj}|dkrtd|j|dkr|S|jd}t|d|g}|S)z@Reshapes a >= rank 2 tensor to a rank 2 tensor (i.e., a matrix).r5z2Input tensor must have at least rank 2. Shape = %sr_)rGrrr&r)rSrrrrrrrs rcCs>t|dkr|St|}|dd}|d}t|||gS)z?Reshapes a rank 2 tensor back to its original rank >= 2 tensor.r5rr_)rrfr&r)rZorig_shape_list output_shapeZ orig_dimsrrrrrs  rcCsx|dur|j}i}t|tjrd||<n |D]}d||<q|jj}||vr:tj}td|||t |jt |fdS)aQRaises an exception if the tensor rank is not of the expected rank. Args: tensor: A tf.Tensor to check the rank of. expected_rank: Python integer or list of integers, expected rank. name: Optional name of the tensor for the error message. Raises: ValueError: If the expected shape doesn't match the actual shape. NTzkFor the tensor `%s` in scope `%s`, the actual rank `%d` (shape = %s) is not equal to the expected rank `%s`) rrr integer_typesrGrr&get_variable_scoperstr)rrDrZexpected_rank_dictr actual_rank scope_namerrrrs      rr)r )rr rLF) FNr rQTrRr r r ) NrEr NNNrBr FNNN)NN)&r? __future__rrrrr1r+rrnumpyrr tensorflowr&tensorflow.compat.v1rrdisable_v2_behaviorobjectrrAr rqrrrrryrkrnrorrprfrrrrrrrs   L"%    1 `$ Y  "%