内容简介

文本处理是目前互联网内容应用(如搜索引擎、推荐引擎)的关键技术。本书涵盖了文本处理概念和技术的多个方面，包括文本预处理、搜索、字符串匹配、信息抽取、命名实体识别、分类、聚类、标签生成、摘要、问答等。本书的特点在于通过实例来理解文本处理的这些概念和技术，读者利用现有的开源工具就可以自己实现这些实例。

作品目录

第1章　开始驾驭文本 ...............................................................................1

1.1　驾驭文本重要的原因 ...............................................................................................2

1.2　预览：一个基于事实的问答系统 ...........................................................................4

1.2.1　嗨，弗兰肯斯坦医生 ...................................................................................5

1.3　理解文本很困难 .......................................................................................................8

1.4　驾驭的文本 .............................................................................................................11

1.5　文本及智能应用：搜索及其他 .............................................................................13

1.5.1　搜索和匹配 .................................................................................................13

1.5.2　抽取信息 .....................................................................................................14

1.5.3　对信息分组 .................................................................................................15

1.5.4　一个智能应用 .............................................................................................15

1.6　小结 .........................................................................................................................15

1.7　相关资源 .................................................................................................................16

第2章　驾驭文本的基础 ..........................................................................17

2.1　语言基础知识 .........................................................................................................18

2.1.1　词语及其类别 .............................................................................................19

2.1.2　短语及子句 .................................................................................................20

2.1.3　词法 .............................................................................................................21

2.2　文本处理常见工具 .................................................................................................23

2.2.1　字符串处理工具 .........................................................................................23

2.2.2　词条及切词 .................................................................................................23

2.2.3　词性标注 .....................................................................................................25

2.2.4　词干还原 .....................................................................................................27

2.2.5　句子检测 .....................................................................................................29

2.2.6　句法分析和文法 .........................................................................................31

2.2.7　序列建模 .....................................................................................................33

2.3　从常见格式文件中抽取内容并做预处理 .............................................................34

2.3.1　预处理的重要性 .........................................................................................35

2.3.2　利用Apache Tika抽取内容 ........................................................................37

2.4　小结 .........................................................................................................................39

2.5　相关资源 .................................................................................................................40

第3章　搜索 ............................................................................................41

3.1　搜索和多面示例：Amazon.com ............................................................................42

3.2　搜索概念入门 .........................................................................................................44

3.2.1　索引内容 .....................................................................................................45

3.2.2　用户输入 .....................................................................................................47

3.2.3　利用向量空间模型对文档排名 .................................................................51

3.2.4　结果展示 .....................................................................................................54

3.3　Apache Solr搜索服务器介绍 .................................................................................57

3.3.1　首次运行Solr ..............................................................................................58

3.3.2　理解Solr中的概念 ......................................................................................59

3.4　利用Apache Solr对内容构建索引 .........................................................................63

3.4.1　使用XML构建索引 ....................................................................................64

3.4.2　利用Solr和Apache Tika对内容进行抽取和索引 ......................................66

3.5　利用Apache Solr来搜索内容 .................................................................................69

3.5.1　Solr查询输入参数 ......................................................................................71

3.5.2　抽取内容的多面展示 .................................................................................74

3.6　理解搜索性能因素 .................................................................................................77

3.6.1　数量判定 .....................................................................................................77

3.6.2　判断数量 .....................................................................................................81

3.7　提高搜索性能 .........................................................................................................82

3.7.1　硬件改进 .....................................................................................................82

3.7.2　分析的改进 .................................................................................................83

3.7.3　提高查询性能 .............................................................................................85

3.7.4　其他评分模型 .............................................................................................88

3.7.5　提升Solr性能的技术 ..................................................................................89

3.8　其他搜索工具 .........................................................................................................91

3.9　小结 .........................................................................................................................93

3.10　相关资源 ...............................................................................................................93

第4章　模糊字符串匹配 ..........................................................................94

4.1　模糊字符串匹配方法 .............................................................................................96

4.1.1　字符重合度度量方法 .................................................................................96

4.1.2　编辑距离 .....................................................................................................99

4.1.3　n元组编辑距离 .........................................................................................102

4.2　寻找模糊匹配串 ...................................................................................................105

4.2.1　在Solr中使用前缀来匹配 ........................................................................105

4.2.2　利用trie树进行前缀匹配 .........................................................................106

4.2.3　使用n元组进行匹配 ..................................................................................111

4.3　构建模糊串匹配应用 ...........................................................................................112

4.3.1　在搜索中加入提前输入功能 ...................................................................113

4.3.2　搜索中的查询拼写校正 ...........................................................................117

4.3.3　记录匹配 ...................................................................................................122

4.4　小结 .......................................................................................................................127

4.5　相关资源 ...............................................................................................................128

第5章　命名实体识别 ...........................................................................129

5.1　命名实体的识别方法 ...........................................................................................131

5.1.1　基于规则的实体识别 ...............................................................................131

5.1.2　基于统计分类器的实体识别 ...................................................................132

5.2　基于OpenNLP的基本实体识别 ...........................................................................133

5.2.1　利用OpenNLP寻找人名 ...........................................................................134

5.2.2　OpenNLP识别的实体解读 .......................................................................136

5.2.3　基于概率过滤实体 ...................................................................................137

5.3　利用OpenNLP进行深度命名实体识别 ...............................................................137

5.3.1　利用OpenNLP识别多种实体类型 ...........................................................138

5.3.2　OpenNLP识别实体的背后机理 ...............................................................141

5.4　OpenNLP的性能 ...................................................................................................143

5.4.1　结果的质量 ...............................................................................................144

5.4.2　运行性能 ...................................................................................................145

5.4.3　OpenNLP的内存使用 ...............................................................................146

5.5　对新领域定制OpenNLP实体识别 .......................................................................147

5.5.1　训练模型的原因和方法 ...........................................................................147

5.5.2　训练OpenNLP模型 ...................................................................................148

5.5.3　改变建模输入 ...........................................................................................150

5.5.4　对实体建模的新方法 ...............................................................................152

5.6　小结 .......................................................................................................................154

5.7　进一步阅读材料 ...................................................................................................155

第6章　文本聚类 ..................................................................................156

6.1　Google News中的文档聚类 .................................................................................157

6.2　聚类基础 ...............................................................................................................158

6.2.1　三种聚类的文本类型 ...............................................................................158

6.2.2　选择聚类算法 ...........................................................................................160

6.2.3　确定相似度 ...............................................................................................161

6.2.4　给聚类结果打标签 ...................................................................................162

6.2.5　聚类结果的评估 .......................................................................................163

6.3　搭建一个简单的聚类应用 ...................................................................................165

6.4　利用Carrot2对搜索结果聚类 ...............................................................................166

6.4.1　使用Carrot2API ........................................................................................166

6.4.2　使用Carrot2对Solr的搜索结果聚类 ........................................................168

6.5　利用Apache Mahout对文档集聚类 ......................................................................171

6.5.1　对聚类的数据进行预处理 .......................................................................172

6.5.2　K-means聚类 ............................................................................................175

6.6　利用Apache Mahout进行主题建模 ......................................................................180

6.7　考察聚类性能 .......................................................................................................183

6.7.1　特征选择与特征约简 ...............................................................................183

6.7.2　Carrot2的性能和质量 ...............................................................................186

6.7.3　Mahout基准聚类算法 ..............................................................................187

6.8　致谢 .......................................................................................................................192

6.9　小结 .......................................................................................................................192

6.10　参考文献 .............................................................................................................193

第7章　分类及标注 ...............................................................................195

7.1　分类及归类概述 ...................................................................................................197

7.2　分类过程 ...............................................................................................................200

7.2.1　选择分类机制 ...........................................................................................201

7.2.2　识别文本分类中的特征 ...........................................................................202

7.2.3　训练数据的重要性 ...................................................................................203

7.2.4　评估分类器性能 .......................................................................................206

7.2.5　将分类器部署到生产环境 .......................................................................208

7.3　利用Apache Lucene构建文档分类器 ..................................................................209

7.3.1　利用Lucene对文本进行分类 ...................................................................210

7.3.2　为MoreLikeThis分类器准备训练数据 ....................................................212

7.3.3　训练MoreLikeThis分类器 ........................................................................214

7.3.4　利用MoreLikeThis分类器对文档进行分类 ............................................217

7.3.5　测试MoreLikeThis分类器 ........................................................................220

7.3.6　将MoreLikeThis投入生产环境 ................................................................223

7.4　利用Apache Mahout训练朴素贝叶斯分类器 ......................................................223

7.4.1　利用朴素贝叶斯算法进行文本分类 .......................................................224

7.4.2　准备训练数据 ...........................................................................................225

7.4.3　留存测试数据 ...........................................................................................229

7.4.4　训练分类器 ...............................................................................................229

7.4.5　测试分类器 ...............................................................................................231

7.4.6　改进自举过程 ...........................................................................................232

7.4.7　将Mahout贝叶斯分类器集成到Solr ........................................................234

7.5　利用OpenNLP进行文档分类 ...............................................................................238

7.5.1　回归模型及最大熵文档分类 ...................................................................239

7.5.2　为最大熵文档分类器准备训练数据 .......................................................241

7.5.3　训练最大熵文档分类器 ...........................................................................242

7.5.4　测试最大熵文档分类器 ...........................................................................248

7.5.5　生产环境下的最大熵文档分类器 ...........................................................249

7.6　利用Apache Solr构建标签推荐系统 ...................................................................250

7.6.1　为标签推荐收集训练数据 .......................................................................253

7.6.2　准备训练数据 ...........................................................................................255

7.6.3　训练Solr标签推荐系统 ............................................................................256

7.6.4　构建推荐标签 ...........................................................................................258

7.6.5　对标签推荐系统进行评估 .......................................................................261

7.7　小结 .......................................................................................................................263

7.8　参考文献 ...............................................................................................................265

第8章　构建示例问答系统 ....................................................................266

8.1　问答系统基础知识 ...............................................................................................268

8.2　安装并运行QA代码 .............................................................................................270

8.3　一个示例问答系统的架构 ...................................................................................271

8.4　理解问题并产生答案 ...........................................................................................274

8.4.1　训练答案类型分类器 ...............................................................................275

8.4.2　对查询进行组块分析 ...............................................................................279

8.4.3　计算答案类型 ...........................................................................................280

8.4.4　生成查询 ...................................................................................................283

8.4.5　对候选段落排序 .......................................................................................285

8.5　改进系统的步骤 ...................................................................................................287

8.6　本章小结 ...............................................................................................................287

8.7　相关资源 ...............................................................................................................288

第9章　未驾驭的文本：探索未来前沿 ..................................................289

9.1　语义、篇章和语用：探索高级NLP ....................................................................290

9.1.1　语义 ...........................................................................................................291

9.1.2　篇章 ...........................................................................................................292

9.1.3　语用 ...........................................................................................................294

9.2　文档及文档集自动摘要 .......................................................................................295

9.3　关系抽取 ...............................................................................................................298

9.3.1　关系抽取方法综述 ...................................................................................299

9.3.2　评估 ...........................................................................................................302

9.3.3　关系抽取工具 ...........................................................................................303

9.4　识别重要内容和人物 ...........................................................................................303

9.4.1　全局重要性及权威度 ...............................................................................304

9.4.2　个人重要性 ...............................................................................................305

9.4.3　与重要性相关的资源及位置 ...................................................................306

9.5　通过情感分析来探测情感 ...................................................................................306

9.5.1　历史及综述 ...............................................................................................307

9.5.2　工具及数据需求 .......................................................................................308

9.5.3　一个基本的极性算法 ...............................................................................309

9.5.4　高级话题 ...................................................................................................311

9.5.5　用于情感分析的开源库 ...........................................................................312

9.6　跨语言检索 ...........................................................................................................313

9.7　本章小结 ...............................................................................................................315

9.8　相关资源 ...............................................................................................................315