Chinese Weibo sentiment analysis evaluation outline .docx

Chinese Weibo Sentiment Analysis Evaluation OutlineEvaluation ObjectThe object of this evaluation is the core technology of sentiment analysis for Chinese Weibo, including opinion sentence recognition, emotional tendency analysis and emotional element extraction. Task SettingThis evaluation sets up the following 3 sub-tasks, of which Task 1 is a mandatory task, and Task 2 and Task 3 are based on Task 1, and participating teams can choose them. Opinion sentence recognitionFor each sentence in each microblog, this task requires to determine whether the sentence is an opinion sentence or a non-opinion sentence. Submission format: idrun-tagweibo-idsentence-id opinionDescription id: result serial numberrun-tag: team IDWEIBO-id: Weibo idsentence-id: sentence idopinion: opinion sentence identification, Y if it is an opinion sentence, N if it is a non-opinion sentence, for example, the following two Weibo: weibo1: weibo id="1"sentence id="1" The Weinan City Management Tearing Spring Festival couplets incident was reported by Focus Media on the Chengdu bus! /sentence sentence id="2" Weinan City Management can't afford to be hurt! /sentence /weiboweibo2:weibo id="2" sentence id="1" LinkedIn In June 2011, there were about 1 million users in Chinese mainland, and now it is more than 2 million. /sentencesentence id="2" LinkedIn probably hasn't done a penny of advertising, maybe IPO is a very good advertisement.

There are two sentences in /sentence/weiboweibo1, the first is a non-opinion sentence and the second sentence is an opinion sentence. There are 4 sentences in weibo2, of which the first sentence is a non-opinion sentence and the second sentence is an opinion sentence. The correct output is: 1xyz11N2xyz12Y3xyz21N4xyz22Y Note: The definition of opinion sentences in this review does not include sentences that express one's own emotions, such as "I feel happy." Such a sentence is an emotional sentence, but it does not fall under the opinion sentence defined in this review. The opinion sentence defined in this evaluation is limited to the evaluation of other objects, and does not include inner self-feelings. The different fields are separated by \t, the same below. Evaluation CriteriaThis task uses Precision, Recall, and F-measure to evaluate the systems of each participating team. The formula is as follows: #gold是人工标注的结果情感倾向性判断本任务要求判断微博中每条观点句的情感倾向. The emotional tendencies of opinion sentences can be divided into positive, negative, and neutral. Submission Format: idrun-tagweibo-idsentence-id polarityDescription id: result serial numberrun-tag: team IDWEIBO-id: Weibo idsentence-id: Opinion sentence idpolarity: Sentiment propensity identifier, positive is POS, negative is NEG, neutral is NEU.

For example, in the above two Weibo posts weibo1 and weibo2, the second sentence of weibo1 is an opinion sentence, and the emotional tendency is neutral. The second sentence of weibo2 is an opinion sentence, and the emotional tendency is positive. The result should be as follows: 1xyz12NEU2xyz22POS Note: Opinion sentences that cannot be clearly classified into positive or negative should be classified into neutral categories. Evaluation CriteriaThis task also uses Precision, Recall, and F-measure as evaluation criteria. Emotional element extractionThis task requires finding out the evaluation object of the author of each opinion sentence in Weibo, that is, the emotional object. Each opinion sentence may contain several emotional objects. At the same time, the polarity of the point of view for the emotional object is judged. Note: Only the emotional objects in the opinion sentences in Weibo are extracted. This task is a sentence-level task, and the emotional object can only be extracted from the current sentence. For opinion sentences that do not have an emotional object (in some cases the emotional object is implicit), the participating teams may not be drawn. For the definition of the length of the text of the emotional object, this evaluation adopts a flexible approach, such as "Weinan City Management can't afford to hurt!" The object of affection can be: "chengguan" or "weinan chengguan"; However, "Nancheng" or "Nanchengguan" does not belong to the emotional object of this example sentence. In the case where the emotional object is a pronoun, the participating team may or may not do it. For example, "Xiao Ming studied at Peking University, and he was an excellent student. The emotion object can be "he" or "Xiao Ming", submit formatidrun-tagweibo-idsentence-idbegin-offset end-offset polaritydescription id: result serial numberrun-tag: team IDWEIBO-id: Weibo idsentence-id: sentence idbegin-offset: The starting position of the emotional object end-offset: the ending position of the emotional object polarity: the polarity of the opinion on the emotional object, POS represents positive, NEG represents negative, and NEU represents neutral.

For example, the emotional elements of weibo1 and weibo2 are as follows: 1xyz1203NEU2xyz2207NEG3xyz222416 POS Note: Polarity of an opinion that cannot be clearly classified as positive or negative should be classified as neutral. The file is encoded in Unicode (UTF-16), each character occupies two bytes, and the offset of the first character in any sentence is 0, the offset of the second character is 1, and so on. For example, the offsets corresponding to the four characters of "Weinan Chengguan" at the beginning of the second sentence of weibo1 are 0, 1, 2, and 3 respectively. Evaluation criteria: Precision, Recall, and F-measure are also used as evaluation criteria for this task. Evaluation dataset: The evaluation data comes from Tencent Weibo. The complete set of evaluation data includes 20 topics, and each topic collects about 1,000 microblogs, for a total of about 20,000 microblogs. The data is in XML format and the sentences are pre-segmented. Example: weibo1:weibo id="1" sentence id="1" The incident of Weinan city management tearing the Spring Festival couplets was widely reported by Focus Media on the bus in Chengdu! /sentence sentence id="2" Weinan City Management can't afford to be hurt! /sentence /weiboweibo2:weibo id="2" sentence id="1" LinkedIn In June 2011, there were about 1 million users in Chinese mainland, and now it is more than 2 million.

/sentencesentence id="2" LinkedIn probably hasn't done a penny of advertising, maybe IPO is a very good advertisement. /sentence/weibo, where each Weibo corresponds to a weibo element, and each sentence corresponds to a sentence element. The file is encoded in Unicode (UTF-16). Evaluation Method: This evaluation is an offline evaluation. The participating units process the data on their own, generate the corresponding results and submit them. The answer is determined by manual annotation. Entrants need to process all the evaluation data, but the manually annotated data used for the actual evaluation is only about 10% of the complete set of evaluation data. The specific evaluation steps are: the evaluation unit provides test samples (including answers) in advance, the evaluation unit gives the test data, the participating unit runs the tested system, obtains the test results, the participating unit submits the test results, the evaluation unit marks the answers, runs the automatic evaluation program, and counts the evaluation results. The participating system shall pre-train the model and adjust all parameters, and there shall be no manual intervention during operation. This review does not restrict the use of semantic resources. For each sub-task, participants can submit a maximum of 2 sets of results. Evaluation schedule 2012/1/1-2/29: draft evaluation outline and solicit opinions from all parties; 2012/3/1-3/31: Revised and improved the evaluation outline and determined the evaluation data; 2012/4/1: Released the evaluation task and accepted the evaluation registration; 2012/5/1: Released the evaluation sample dataset.

2012/5/2-7/15: Construct evaluation dataset and formulate standard answers; 2012/7/16-7/31: Released the evaluation dataset and submitted the results. 2012/8/1-8/31: Organize an expert evaluation team to evaluate the results and release the evaluation results; 2012/9/1-9/24: Call for evaluation papers; 2012/9/25-9/30: Confirmed invited presentations; 2012/10/29-11/6: Presentation of reports and exchange of experiences and technologies; To register for the Assessment, you will need to download the registration form from the Reviews page below and send it to the Organizer by email or fax before the deadline. Registration should be based on research institutions or companies, and individual applications will not be accepted for the time being. /conference/2012/ If you have any questions about this review, please send an email to: huangxiaojiang@ the organizer of this evaluation, China Computer Federation (CCF), organizer, Peking University, MSRA, co-organizer, State Key Laboratory of Digital Publishing, Evaluation Committee (in alphabetical order of surname): Li Shoushan, Liu Qun, Wan Xiaojun, Wei Furu, Wu Yunfang