elasticsearch|elasticsearch for hadoop
CREATE EXTERNAL TABLE `elasticsearch.t_elastic_xxxx`(
`wayuid` string COMMENT 'from deserializer',
`rpt_date` string COMMENT 'from deserializer',
`start_org_code` string COMMENT 'from deserializer',
`start_org_name` string COMMENT 'from deserializer',
`customer_code` string COMMENT 'from deserializer',
`customer_name` string COMMENT 'from deserializer',
`waybill_no` string COMMENT 'from deserializer',
`order_channel_type` string COMMENT 'from deserializer',
`loading_time` string COMMENT 'from deserializer',
`seller_id` string COMMENT 'from deserializer',
`seller_name` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.elasticsearch.hadoop.hive.EsSerDe'
STORED BY
'org.elasticsearch.hadoop.hive.EsStorageHandler'
WITH SERDEPROPERTIES (
'serialization.format'='1')
LOCATION
'hdfs://xxxxxx/t_fengchao_test'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='false',
'es.batch.size.bytes'='100000000',
'es.batch.size.entries'='100000',
'es.batch.write.refresh'='false',
'es.index.auto.create'='false',
'es.mapping.id'='wayuid',
'es.net.http.auth.pass'='changeme',
'es.net.http.auth.user'='elastic',
'es.nodes'='xxx.xxx.xxx.xxx',
'es.nodes.wan.only'='true',
'es.port'='9200',
'es.resource'='order_no_sign-2020.11.01-1/t_order_no_sign',
'es.write.operation'='upsert')
如果ES使用x-pack后,需要增加
'es.net.http.auth.pass'='changeme',
'es.net.http.auth.user'='elastic',
参数进行权限验证。
【elasticsearch|elasticsearch for hadoop】如果设置为upset,请注意同一批次内_id一定不要有重复数据。
推荐阅读
- 【Hadoop踩雷】Mac下安装Hadoop3以及Java版本问题
- whlie循环和for循环的应用
- ffmpeg源码分析01(结构体)
- 【WORKFOR】最真的自己
- R|R for data Science(六)(readr 进行数据导入)
- performSelectorOnMainThread:withObject:waitUntilDone:参数设置为NO或YES的区别
- Hadoop|Hadoop MapReduce Job提交后的交互日志
- JavaScript|JavaScript — 初识数组、数组字面量和方法、forEach、数组的遍历
- ElasticSearch6.6.0强大的JAVA|ElasticSearch6.6.0强大的JAVA API详解
- Elasticsearch|Elasticsearch 简介